acceptodds
Under review as a conference paper at ICLR 2027

EASE: Effect-Aware Representation Editing for Missing-Modality Prediction

Abstract

Missing modalities typically result in irreversible information loss; consequently, recovering performance as much as possible under these conditions is a compelling research problem. Existing methods address this issue by reconstructing or retrieving missing information, transferring knowledge from modality-complete models, or adapting models to available inputs. These strategies encompass feature alignment, modality experts, adaptive fusion, and prompt learning. However, we aim to explore this from a novel perspective: what impact does a missing modality have on the model's decision-making, and can we restore model performance by recovering that impact? Across 12 settings involving various models and datasets, we find that low-dimensional projections of natural representation shifts between complete and incomplete inputs can substantially preserve the score effects of the unprojected shifts under replay on held-out samples, even when retaining only a small fraction of their hidden-state energy. We propose (Effect-Aware Subspace Editing), a lightweight editor that learns sample-specific corrections from available inputs within a fixed subspace derived from paired complete–incomplete training representations, while keeping the backbone frozen. In this way, we achieve state-of-the-art (SOTA) results on the IEMOCAP and CMU-MOSEI datasets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Missing modalities, Multimodal large language models, Representation editing, Affective computing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TS9eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EASE, a method for handling missing modalities in multimodal large language models that learns sample-specific corrections from available inputs within a fixed low-dimensional space constructed from paired training representations. The authors conduct an analysis across different models and tasks and find that interventions in a low-rank subspace can approximate the label-score changes produced by restoring complete-input representations at the same intervention site. They evaluate the proposed method on three benchmarks and show that it outperforms most baselines across the evaluated settings.

Critical strengths
  1. The authors analyze multiple MLLM backbones and show that low-rank interventions can preserve the decision effects of complete-input representations.

  2. The experimental results are strong, and the proposed method outperforms most baselines across three benchmarks.

  3. The paper is well-written and easy to follow.

Critical weaknesses
  1. The method trains a separate editor for each missing-modality condition, which increases the training cost. However, the authors do not provide a comparison with jointly training a single editor across different missing-modality conditions. It is unclear whether training separate editors is necessary or provides any advantage over joint training.

  2. Although the authors analyze interventions across four MLLM backbones, the proposed EASE method is only trained and evaluated on Emotion-LLaMA-v2. This makes it difficult to assess how well the method generalizes to other backbones.

  3. The choice of intervention layers is not clearly justified. The authors use different layers for different missing-modality conditions but do not explain how these layers were selected or provide an ablation study on the effect of intervention depth. Since the effectiveness of representation editing may depend on the selected layer, this is an important missing analysis.

  4. The text in Figure 2 is very small and difficult to read.

Questions for authors
  1. Although the authors analyze low-rank interventions across four MLLM backbones, EASE is only evaluated on Emotion-LLaMA-v2. How well does the proposed method generalize to other backbones? Without evaluating EASE on additional models, it is difficult to determine whether the improvements are specific to the chosen backbone and intervention layers.

  2. What is the motivation for training a separate editor for each missing-modality condition? Could the authors compare this approach with jointly training a shared editor across different missing-modality conditions to show whether separate training is necessary and justifies the additional training cost?

  3. How are the intervention layers selected for each missing-modality condition? Could the authors clarify the selection criteria and investigate how sensitive the performance is to the choice of intervention layer?

  4. Given that there is existing work that explores low-rank representation editing and knowledge transfer from complete to incomplete modalities, could the authors clarify how EASE differs from combining existing representation-editing methods with conventional knowledge distillation? In particular, what benefit does the proposed effect-aware supervision provide over directly distilling complete-input predictions?

4 comments
AuthorsOpenReview ↗

Response to reviewer TS9e

We thank the reviewer for their thorough and insightful feedback. Below, we address the main concerns and provide additional experimental evidence to substantiate our claims.

  • Q1. Although the authors analyze low-rank interventions across four MLLM backbones, EASE is only evaluated on Emotion-LLaMA-v2. How well does the proposed method generalize to other backbones?

We now train and evaluate EASE on MiniCPM-o 4.5 and AffectGPT, extending the evidence beyond oracle replay on additional backbones.

BackboneDataset / missingFrozenEASEGain
MiniCPM-o 4.5IEMOCAP-4 / V56.9464.90+7.96
MiniCPM-o 4.5IEMOCAP-4 / A56.4161.47+5.06
MiniCPM-o 4.5IEMOCAP-4 / T39.0267.55+28.54
AffectGPTIEMOCAP-4 / T30.3052.48+22.18
AffectGPTMOSI / T47.6256.94+9.32
AffectGPTMOSEI / T39.6871.01+31.33
Table 1. Trained EASE on additional frozen backbones. W-F1 (%); V/A/T denote missing video/audio/text. Gains are relative to the corresponding frozen missing-input baseline, in percentage points.

MiniCPM-o 4.5 gains 13.85 mean W-F1 points across V/A/T. AffectGPT gains 22.18 / 9.32 / 31.33 points under Missing-T on IEMOCAP-4 / MOSI / MOSEI, respectively. These results show that EASE can be trained and deployed on additional frozen backbones using only the available modalities at inference.

  • Q2. What is the motivation for training a separate editor for each missing-modality condition? Could the authors compare this approach with jointly training a shared editor across different missing-modality conditions to show whether separate training is necessary and justifies the additional training cost?

The separate editors used in the paper are one implementation of EASE. They give each missing condition its own recovery basis and predictor. The same framework can also use a shared, condition-aware editor trained jointly across conditions. The two implementations achieve similar mean W-F1 (72.71 vs. 72.59), showing that EASE accommodates both separate and shared editing.

MetricSeparateShared
Missing-V / A / T W-F177.14 / 69.91 / 71.0877.04 / 69.57 / 71.16
Mean W-F172.7172.59
Trainable parameters per fold1,624,914541,638
Training + validation time (hours)107.3764.42
Table 2. Shared versus separate editors. IEMOCAP-4, rank 6, seed 2061; W-F1 (%) averaged over five folds. Parameters are reported per fold, and runtime totals cover all five folds.
  • Q3. How are the intervention layers selected for each missing-modality condition? Could the authors clarify the selection criteria and investigate how sensitive the performance is to the choice of intervention layer?

The intervention depth is preset and fixed across missing conditions. For Emotion-LLaMA-v2, we use the answer-position input of block 23 (relative depth about 0.75), unchanged across datasets and seeds. No test score was used to select this layer. Cross-layer replay on MELD provides supporting evidence:

Layer7152331
Recovered effect (ratio)−0.0020.2560.9180.999
Table 3. Cross-layer full-displacement replay. MELD, 2,610 utterances, dialogue-level five-fold cross-fitting. Each layer replays hfull−hmissingh_{\mathrm{full}}-h_{\mathrm{missing}}; the metric measures score-effect recovery.

Replay recovery depends strongly on depth: layer 23 is the earliest tested layer to recover most of the effect (91.8%), while leaving roughly the final quarter of the network to process the edit. This supports the preset choice.

Figure readability. We will enlarge the text and improve the layout of Figure 2 in the revision.

  • Q4. Could the authors clarify how EASE differs from combining existing representation-editing methods with conventional knowledge distillation? In particular, what benefit does the proposed effect-aware supervision provide over directly distilling complete-input predictions?

EASE uses low-rank editing to recover task-useful modality effects. Its REP carrier is constructed from paired complete-minus-missing hidden-state displacements, providing a compact space of modality-related changes. The effect-aware objective then determines which reference effects the editor should learn to reproduce.

The key distinction from editing + KD is selective supervision. A conventional combination trains a low-rank editor to make incomplete-input predictions resemble complete-input predictions. Although ground-truth CE is also retained, the KD term still encourages agreement with the complete-input model, including on samples where that reference is unhelpful.

EASE instead uses the missing modality’s score effect as a reference. It matches the centered reference only when that correction yields lower CE than the current edit; otherwise, it penalizes excessive changes. Ground-truth CE remains active throughout. This balances recovery of modality effects with task correctness, since complete input does not always produce a better prediction.

We isolate this distinction using the same low-rank editor. Both methods use the same rank-6 REP editor, frozen backbone, data, and validation-selection protocol; only the training objective changes.

SeedLow-rank editing + KDEASEPaired gain
206171.8772.71+0.84
206271.8972.87+0.98
206371.2472.41+1.17
Mean71.6772.66+1.00
Table 4. EASE versus low-rank editing + KD on IEMOCAP-4, using frozen Emotion-LLaMA-v2.

EASE improves W-F1 at all three seeds, by 1.00 point on average. Because the editing architecture is identical, these gains support the benefit of effect-aware supervision over complete-input KD within the same low-rank editing framework.

  • Q3. Layer sensitivity

We also retrained EASE at blocks 28 and 31 to test whether moving the editor makes task performance change.

DatasetBlock 23 (default)Block 28Block 31
IEMOCAP-472.6071.8672.12
MOSI79.5879.7679.71
Table 5. Test W-F1 (%) with the same seed and hyperparameters at each layer, averaged over Missing-V/A/T and, for IEMOCAP-4, five folds.*

Performance changes little across these tested late layers: W-F1 varies by 0.74 points on IEMOCAP-4 and 0.18 on MOSI. Higher replay recovery at deeper layers does not consistently improve task performance, as we proposed in Section 3.

Thank you for the additional experiments and clarifications. They address most of my concerns. The comparison with standard distillation also supports the benefit of supervision, although the gain is modest and has only been tested on one dataset. I will maintain my original rating of 3.

Reviewer_7v8HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EASE, a lightweight representation editor for prediction with a missing modality. The authors first use truncated SVD of complete-minus-incomplete hidden-state differences to construct a low-rank REP subspace and study whether it preserves downstream score effects. EASE then predicts sample-specific coefficients in this fixed subspace while keeping the MLLM backbone frozen. Its objective combines task supervision, utility-gated effect matching, and magnitude regularization. Experiments on IEMOCAP-4, CMU-MOSI, and CMU-MOSEI cover Missing-V/A/T and show improvements over the frozen backbone in all nine settings, with additional same-backbone editing controls.

Critical strengths
  • The paper carefully distinguishes representation difference, score effect, and classification utility, avoiding the common assumption that a more faithful reconstruction must yield a better prediction.
  • The diagnostic study covers four MLLMs and three main datasets, with additional MELD and MER2023 results.
  • Rank-6 REP obtains effect NMSE between 0.052 and 0.302, while random subspaces remain near 1, providing strong evidence that the low-rank carrier captures relevant effect directions.
  • The paper includes held-out diagnostics and controls using mean, shuffled, orthogonal, JAC, ReFT/LoReFT, task-only, and learned carriers.
  • The authors report counterexamples: better effect fidelity does not always improve classification, and sample-specific coefficients provide only weak evidence on MOSI.
  • The backbone is frozen, the editor is small, and several comparisons include paired bootstrap confidence intervals.
Critical weaknesses
  1. The independent contribution of effect matching is not identified. At minimum, the paper needs CE + magnitude, matching + magnitude without the gate, and controls that use random or shuffled reference directions under the same regularization strength.
  2. The trainable EASE editor is evaluated with only one backbone. Results across four models are primarily oracle replay using the true complete-incomplete displacement. The deployable editor is trained only with Emotion-LLaMA-v2.

Besides, I also have few concerns:

  • The six external methods in Table 2 use different backbones, features, and training pipelines, so these comparisons are descriptive rather than controlled.
  • LearnedCarrier-CE and EASE change both the carrier and the training objective, so this comparison does not isolate the contribution of REP. Under CE-only training on IEMOCAP, the learned carrier's average is slightly better than REP task-only.
  • Appendix A.2 shows that replaying the full hidden displacement at one layer still has 0.305 average NMSE relative to the true complete-input effect, reaching 0.776 for Missing-A. The bridge between the diagnostic and the final training target remains heuristic.
  • Most main experiments use a single training seed. Test-sample bootstrap intervals with fixed checkpoints do not capture training variance or hyperparameter-selection uncertainty.
  • The complete intervention layer, lambda, radius quantile, and selection protocol are not fully reported for the main rank-3 IEMOCAP and MOSI/MOSEI configurations.
  • The method requires a separate editor for every known missing condition and paired complete training data. Simultaneous missing modalities and naturally occurring missingness are not evaluated.
  • The reference effect is centered by subtracting the training mean, but that mean is not restored at inference. There is no uncentered control or report of the gate activation rate.
Questions for authors
  1. Can you add CE + magnitude and matching + magnitude without the utility gate?
    • If effect matching still gives a stable, multi-seed improvement once magnitude regularization is already present, I would raise my overall score.
  2. Can you train EASE, rather than only perform oracle replay, on Qwen2.5-Omni, AffectGPT, or MiniCPM-o?
  3. Can you compare REP, JAC, random, activation-PCA, and learned carriers under exactly the same training objective?
  4. How were the intervention layer, lambda, and radius quantiles selected for the main configurations? Was the test set completely untouched?
  5. Why is centering the reference effect necessary? What happens when directly matching D_i,m or adding the mean back at inference?
4 comments
AuthorsOpenReview ↗

Response to reviewer 7v8H

Thank you for your profound insights and reflections! Thank you for your profound insights and reflections!

  • Q1.Can you add CE + magnitude and matching + magnitude without the utility gate?

Thank you for highlighting the importance of isolating the contribution of effect matching. To address this concern, we conducted additional experiments with CE + magnitude and matching + magnitude across multiple seeds. As shown in Tables 1 and 2, effect matching consistently improves performance on almost all three missing conditions on both IEMOCAP-4 and CMU-MOSI.

Without the gate, matching improves IEMOCAP-4 Missing-A/T and MOSI Missing-V/A at all three seeds. EASE’s gated objective improves IEMOCAP-4 Missing-V and MOSI Missing-T from 76.11 to 77.06 and 61.18 to 61.83, respectively. The gate lets EASE learn from the reference when it helps; otherwise, EASE limits overly large edits without forcing a match.

ObjectiveMissing-VMissing-AMissing-TMean
CE + magnitude76.5568.2170.6271.79
matching + magnitude76.1169.0771.2872.16
EASE77.0669.4071.5372.66
Table 1. Ablation study of effect matching on IEMOCAP-4, reporting mean W-F1 (%) across three random seeds (2061–2063).
ObjectiveMissing-VMissing-AMissing-TMean
CE + magnitude89.2886.7261.6779.22
matching + magnitude89.6587.3661.1879.39
EASE89.5387.6661.8379.67
Table 2. Effect-matching ablations on CMU-MOSI. W-F1 (%) averaged over three seeds (2061–2063).
Condition206120622063Mean gain
Missing-V−0.50−0.28−0.54−0.44
Missing-A+1.29+0.57+0.73+0.86
Missing-T+0.74+0.40+0.85+0.66
Mean+0.51+0.23+0.35+0.36
Table 1.2. Per-seed gains from effect matching on IEMOCAP-4.
Condition206120622063Mean gain
Missing-V+0.33+0.17+0.61+0.37
Missing-A+0.57+0.61+0.75+0.64
Missing-T−1.09−0.45+0.07−0.49
Mean−0.06+0.11+0.48+0.17
Table 2.2. Per-seed gains from effect matching on CMU-MOSI.
  • Q2. Can you train EASE, rather than only perform oracle replay, on Qwen2.5-Omni, AffectGPT, or MiniCPM-o?

We trained and evaluated EASE on two additional backbones, extending the evidence beyond oracle replay. With MiniCPM-o 4.5, EASE improves all three missing-modality conditions on IEMOCAP-4, gaining 5.06–28.53 W-F1 points. With AffectGPT, it improves Missing-T performance on all three datasets, gaining 9.20–31.33 points.

These results show that the learned EASE editor’s benefits extend beyond Emotion-LLaMA-v2, with improvements across both additional backbones and all evaluated settings.

MethodMissing-VMissing-AMissing-T
Frozen56.9456.4139.02
EASE64.90 (+7.96)61.47 (+5.06)67.55 (+28.53)
Table 3. Trained EASE on MiniCPM-o 4.5. W-F1 (%) on IEMOCAP-4 across three missing-modality conditions. Parentheses report gains over the frozen backbone, Δ=W-F1EASE−W-F1Frozen\Delta = \text{W-F1}_{\text{EASE}} - \text{W-F1}_{\text{Frozen}} , in percentage points.
MethodIEMOCAP-4CMU-MOSICMU-MOSEI
Frozen30.3047.6439.68
EASE52.48 (+22.18)56.84 (+9.20)71.01 (+31.33)
Table 4. Trained EASE on AffectGPT. W-F1 (%) under Missing-T across three datasets. Parentheses report gains over the frozen backbone in percentage points.
  • Q3.Can you compare REP, JAC, random, activation-PCA, and learned carriers under exactly the same training objective?
CarrierMissing-VMissing-AMissing-T
REP (ours)77.7568.4371.63
Activation PCA77.5568.5771.76
JAC76.3768.8071.29
Learned76.2968.9271.17
Random74.3466.7170.76
Table 5. Comparison of five carriers under the same EASE training objective. W-F1 (%) on IEMOCAP-4 with frozen Emotion-LLaMA-v2. All carriers have rank 6. Results are averaged over five folds with seed 2061. The best result in each column is highlighted in bold.

EASE works across structured carriers. Under the same objective, all four structured carriers outperform random in every missing-modality condition. REP leads on Missing-V and remains competitive on Missing-A/T. REP provides a compact, fixed carrier, while EASE’s central methodological contribution is learning task-useful edits through effect-aware supervision.

Why REP. REP combines competitive performance with a simple construction tailored to missing-modality shifts. Truncated SVD of paired training displacements, hfull−hmissingh_{\mathrm{full}}-h_{\mathrm{missing}}, extracts a compact basis directly from the representation changes we aim to recover. The basis is computed once per training split and missing condition, then kept fixed, requiring neither class-score Jacobians nor additional trainable basis parameters. This makes REP a practical and computationally convenient choice.

  • Q4. How were the intervention layer, λ\lambda, and radius quantiles selected for the main configurations? Was the test set completely untouched?

We clarify the selection of the intervention layer, hyperparameters, and evaluation protocol as follows.

Layer Intervention and sensitivity. We selected the intervention depth using cross-layer replay on MELD , independently of the evaluation benchmarks. Among the tested layers , layer 23 was the earliest to recover most of the missing-modality effect (91.8%), while preserving the final quarter of the network for subsequent processing.

Layer7152331
Recovered effect (ratio)−0.0020.2560.9180.999
Table 6. Cross-layer full-displacement replay

We also retrained EASE at blocks 28 and 31 to test whether moving the editor makes task performance change.

DatasetBlock 23 (default)Block 28Block 31
IEMOCAP-472.6071.8672.12
MOSI79.5879.7679.71
Table 7. Test W-F1 (%) with the same seed and hyperparameters at each layer, averaged over Missing-V/A/T and, for IEMOCAP-4, five folds.

Performance changes little across these tested late layers: W-F1 varies by 0.74 points on IEMOCAP-4 and 0.18 on MOSI. Higher replay recovery at deeper layers does not consistently improve task performance, as we proposed in Section 3.

Hyperparameters. For IEMOCAP-4, the main configurations use λ=0.25\lambda = 0.25 and CAP radius quantiles of q=0.95/0.50/0.95q = 0.95/0.50/0.95 for Missing-V/A/T, respectively. Separate validation-only sensitivity analyses examine λ∈{0.10,0.25,0.50}\lambda \in \{0.10, 0.25, 0.50\} and Missing-A quantiles q∈{0.50,0.75,0.95}q \in \{0.50, 0.75, 0.95\}. For MOSI, λ\lambda is fixed at 0.100.10.

Data splits and test integrity. Test sets are never used for layer selection, hyperparameter tuning, or checkpoint selection, but only for final evaluation. IEMOCAP uses five-fold session-held-out evaluation, with dialogue-disjoint training and validation splits shared across all objectives and seeds. MOSI and MOSEI use their official train/validation/test splits. SVD bases and reference statistics are computed from training data only. Validation data are used for checkpoint selection (up to 100 epochs, patience 10) and sensitivity analyses.

  • Q5. Why is centering the reference effect necessary? What happens when directly matching D_i,m or adding the mean back at inference?

Why center the reference effect? We subtract the average effect of each missing condition, di,m=Di,m−μmd_{i,m}=D_{i,m}-\mu_m, so matching focuses on differences between samples. CE still supervises the final predictions and can learn useful shared corrections. The results below show modest improvements from centering, although it is not strictly required for effective matching.

DatasetMatching to Di,mD_{i,m}Matching to Di,m−μmD_{i,m}-\mu_m (EASE)
IEMOCAP72.7072.71 (+0.01)
MOSEI83.3083.59 (+0.29)
Table 8. W-F1 (%) using the official readout; parentheses show gains in percentage points. Gate and CAP statistics are recomputed for the uncentered control.*

Why not add the mean back at inference? The output scores have never had the mean subtracted, so there is no corresponding step to reverse. Adding QμmQ\mu_m instead applies an extra fixed shift to the class scores, which can change a correct prediction into a wrong one. This changes W-F1 by −0.39±0.27-0.39 \pm 0.27 on IEMOCAP and +0.05+0.05 on MOSEI, showing no consistent benefit.

Missing conditionFirst epochValidation-selected epoch
Missing-V64.0%22.7%
Missing-A68.0%46.0%
Missing-T67.7%39.3%
Table 9. Approximate mean gate activation on IEMOCAP training samples across seeds 2061–2063.*

The gate balances effect recovery with control of unnecessary changes. The gate turns on matching only when the reference would do better than the editor’s current edit. In the first epoch this is true for about two thirds of the samples (64–68%). By the selected epoch it is true for fewer (23–46%), mostly those where the complete input fixes a missing-input error, and CAP keeps the remaining edits small.

AuthorsOpenReview ↗

Response for other concerns

  • The six external methods in Table 2 use different backbones, features, and training pipelines, so these comparisons are descriptive rather than controlled.

Same-backbone comparisons with editing baselines. Alongside the external comparisons in Table 2, we evaluate LoReFT and ReFT using the same backbone, data splits, intervention site, and validation-selection protocol as EASE. On IEMOCAP-4 with seed 2061, EASE achieves 72.71 mean W-F1, compared with 71.51 for LoReFT (CE) and 70.96 for ReFT (CE). These results compare the methods under a common experimental setup; the same-editor objective ablations isolate the contribution of effect matching.

  • Appendix A.2 shows that replaying the full hidden displacement at one layer still has 0.305 average NMSE relative to the true complete-input effect, reaching 0.776 for Missing-A. The bridge between the diagnostic and the final training target remains heuristic.

Clarifying the replay error and training target. The Missing-A NMSE of 0.776 was measured at block 15. At block 23, where EASE edits, full-displacement replay yields NMSE of 0.09 / 0.10 / 0.05 for Missing-V/A/T. Replay at the actual editing layer therefore reproduces the complete-input score effect much more closely. The training reference is computed directly from complete- and missing-input scores and then centered; it does not use the replay approximation as its target.

  • The method requires a separate editor for every known missing condition and paired complete training data. Simultaneous missing modalities and naturally occurring missingness are not evaluated.

Additional evaluation on simultaneous missingness and degraded inputs. We evaluate the trained editors in two additional settings, without retraining.

First, we remove two modalities simultaneously and reuse the editors trained for single-modality missingness through three composition rules. All three rules improve on the frozen backbone in every condition.

MethodText onlyAudio onlyVideo only
Frozen backbone47.8650.5228.60
Sum of separate-editor edits56.5755.8441.47
Sum of shared-editor coefficients58.9160.9238.76
Shared editor with averaged condition embeddings63.0864.5739.09
Table 10. Two modalities missing simultaneously. IEMOCAP-4, W-F1 (%) from pooled five-fold predictions. Each column indicates the only remaining modality. Editors trained under single-modality missingness are reused without retraining.

Second, we test inputs where video or audio remains available but its quality has deteriorated. EASE also improves performance under both degradation conditions.

Input degradationFrozenEASEGain
Video at 1/8 resolution66.6275.44+8.82
Audio with white noise at 0 dB SNR68.1772.69+4.52
Table 11. EASE under input degradation. IEMOCAP-4, pooled test W-F1 (%). Editors are applied without retraining.

Complete paired inputs provide supervision during training; inference uses only the available inputs.

Reviewer_eyNKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EASE, an Effect-Aware Subspace Editing framework that learns sample-specific corrections from available inputs in a fixed latent space while keeping the backbone frozen. The authors first analyze how missing-modality effects can be captured in a latent space derived using SVD. EASE trains an editor (i.e., a predictor) on paired complete and incomplete training representations to predict sample-specific interventions from incomplete inputs. The edited hidden representations are then used for downstream tasks.

Experiments on three datasets show that EASE outperforms existing baselines across different missing-modality scenarios.

Critical strengths

S1: Editing hidden representations under missing-modality conditions is a well-motivated and practically meaningful research direction.

S2: The proposed method shows promising improvements over the baselines reported in the paper.

S3: The proposed editor is lightweight and keeps the backbone frozen.

Critical weaknesses
  • W1: The rationale for using the difference hi,full−hi,mh_{i,full}-h_{i,m} is unclear, particularly because hh is a high-dimensional vector whose entries can be positive or negative. The authors should justify this choice over alternative distance measures, as well as the use of SVD over other dimensionality-reduction or projection strategies. Can EASE handle more than one missing modality?

  • W2: The equations and notation require further explanation. For example, what does bmb_m represent in Eq. 7? How is NMSE calculated for vi,mMEANv_{i,m}^{MEAN} and vi,mSHUFFLEDv_{i,m}^{SHUFFLED} in Table 4?

  • W3: The experimental evaluation is limited. Ablations of key components, such as the effect-matching loss and regularization, together with sensitivity analyses of the hyper-parameters in Eq. 21, would better support the model design.

  • W4: Visualizations of representative success and failure cases would help demonstrate the effectiveness of representation editing. What happens when the editor generates an incorrect representation?

Questions for authors

Q1: What motivates the use of subtraction and SVD to construct the latent space?

Q2: Could the authors clarify the equations and notation in the paper?

Q3: Could the authors provide ablations of key model components and hyperparameter sensitivity analyses?

Q4: Could the authors provide visualizations of representative success and failure cases?

5 comments
AuthorsOpenReview ↗

Response to reviewer eyNK

We thank the reviewer for their insightful reflections and the suggestions offered regarding our paper!

Q1. What motivates the use of subtraction and SVD to construct the latent space?

Motivation for paired differences. EASE aims to recover the decision-relevant representational shifts caused by missing modalities. We therefore use the difference between complete- and incomplete-input hidden states, as subtraction preserves both the direction and magnitude of these shifts, unlike scalar distance measures that discard directional information.

Our preliminary experiments confirm the importance of direction. Incorporating directional information improves the prediction of decision-effect magnitude (additional R2R^2: 0.196–0.548) and the identification of the most influential missing modality (+7.3–32.9 percentage points over magnitude-only baselines). Furthermore, Table 1 shows that direction-aligned interventions produce substantially larger decision effects than orthogonal interventions, even when edit magnitudes are identical. These findings demonstrate that the direction of a representation shift carries decision-relevant information beyond its magnitude.

Model–datasetAligned − orthogonal effect95% CI
Qwen–MELD+1.235[1.115, 1.362]
AffectGPT–IEMOCAP+3.720[3.121, 4.380]
MiniCPM–MER2023+2.026[1.809, 2.248]

*Table 1. Comparison of direction-aligned and orthogonal interventions at equal edit magnitudes.

Motivation for SVD. We apply SVD to extract the dominant directions shared across training displacements, constructing a compact recovery space without additional basis training. SVD provides an optimal low-rank approximation in terms of squared reconstruction error. Our preliminary analysis (Section 3) further shows that the resulting subspace preserves decision-level effects substantially better than a random subspace (NMSE: 0.052–0.302 vs. 0.988–1.003). NMSE is the squared error between replayed and original score changes, normalized by the squared magnitude of the original changes.

MeasureSVD space (REP)Random space
Error in reproducing the original score change (NMSE ↓\downarrow)0.052–0.3020.988–1.003
Retained variation across examples after mean removal (↑\uparrow)23.73–51.28%0.143–0.168%

*Table 2. Comparison of SVD-based and random low-rank subspaces. NMSE measures the reconstruction error of decision-level score changes, while retained variation measures the proportion of centered displacement variation preserved by each subspace.

To evaluate alternative projection strategies, we additionally compare five carrier constructions under the identical EASE training objective.

CarrierMissing-VMissing-AMissing-T
REP (ours)77.7568.4371.63
Activation PCA77.5568.5771.76
JAC76.3768.8071.29
Learned76.2968.9271.17
Random74.3466.7170.76

Table 3. Comparison of five carrier constructions under the same EASE objective on IEMOCAP-4. Results report W-F1 (%) averaged over five folds using frozen Emotion-LLaMA-v2. Best results are highlighted in bold.

All four structured carriers outperform the random baseline, while REP achieves competitive performance across all missing conditions. Unlike Jacobian-based or learned carriers, REP requires neither Jacobian computation nor additional basis optimization. Thus, SVD-based REP offers a simple and efficient recovery space, although EASE is not limited to this particular construction.

Q1(b). Can EASE handle more than one missing modality?

Yes, without retraining. We took the trained single-missing editors and applied them to inputs with two modalities removed. We combined them with three rules, all fixed before evaluation:

  • (i) Add the edits of the two corresponding EASE editors.
  • (ii) Apply the shared editor once per missing modality and add its coefficients.
  • (iii) Apply the shared editor once with the mean of the two condition embeddings.
MethodText onlyAudio onlyVideo only
Frozen backbone47.8650.5228.60
(i) Sum of separate-editor edits56.5755.8441.47 (+12.87)
(ii) Sum of shared-editor coefficients58.9160.9238.76
(iii) Shared editor, mean condition embedding63.08 (+15.22)64.57 (+14.05)39.09

Table 4. Two missing modalities: reuse without retraining. Results report W-F1 (%); column headings indicate the remaining modality. Best results are highlighted in bold, with gains over the frozen backbone in parentheses.

Removing two modalities is far more damaging than removing one: the frozen backbone drops from 75.98 with complete input to 47.86 / 50.52 / 28.60 W-F1. Every composition rule improves on it in every setting. The best rule adds +15.2 / +14.1 / +12.9 points, closing 54% / 55% / 27% of the gap to complete input, although the editors never saw a two-missing input during training.

With only text or only audio remaining, the edited model (63.08 / 64.57) reaches the range the frozen backbone attains with just one modality missing (60.20–65.76). These results show that the single-missing editors can be composed to handle two missing modalities.

Q2: Could the authors clarify the equations and notation in the paper?

Definition of bmb_m. Eq. (6), lines 180–184, defines bmb_m as the training-set mean of the paired hidden displacement δhi,m=hi,full−hi,m\delta h_{i,m}=h_{i,\mathrm{full}}-h_{i,m}. MEAN applies its projection, Pm,kbmP_{m,k}b_m, as the same edit for every sample under missing condition mm.

NMSE for MEAN and SHUFFLED. Eqs. (7)–(8), lines 186–193, define the controls: MEAN uses the projected training mean, while SHUFFLED uses another sample’s projected displacement within the same missing condition and construction fold, with no self-matches. Their NMSE is computed using Eq. (5), lines 146–151, replacing the matched edit with the corresponding control edit. Specifically, we sum the squared differences between the control’s effect and the current sample’s own full-displacement replay effect, fi,m(δhi,m)f_{i,m}(\delta h_{i,m}), and divide by the summed squared norm of that reference effect. Both controls therefore use the same reference and denominator, consistent with the Table 4 caption (lines 766–767). We will add these cross-references beside Table 4.

Q3. Could the authors provide ablations of key model components and hyperparameter sensitivity analyses?

Full EASE achieves the highest mean W-F1 on both datasets. The table separates the contributions of effect matching, magnitude control (CAP), and utility selection.

DatasetMatchingCAPGateMissing-VMissing-AMissing-TMean
IEMOCAP-4✗✓✗76.8668.7270.3471.97
✓✓✗76.3670.0171.0872.48
✗✓✓75.5868.3970.2671.41
✓✓✓77.1469.9171.0872.71
MOSI✗✗✗88.6986.7660.8978.78
✓✗✓89.0286.8661.6279.17
✗✓✓89.1786.8861.3679.14
✓✓✓89.4887.6262.3179.80

Table 5. Component ablations on IEMOCAP-4 and CMU-MOSI. Results report W-F1 (%); ✓/✗ indicate active/inactive components. Bold marks the best result within each dataset.

Matching adds value beyond magnitude control. With CAP and the gate retained, adding matching improves all three missing conditions on both datasets. On MOSI, removing CAP while retaining gated matching also lowers all three scores, supporting their complementary roles.

The gate helps matching focus on useful corrections. On IEMOCAP Missing-V, adding the gate to matching + CAP raises W-F1 from 76.36 to 77.14. With the gate retained, adding matching raises it from 75.58 to 77.14. These comparisons support combining effect matching with utility selection.

Hyperparameter sensitivity. We evaluated five (λ,q)(\lambda,q) settings per missing condition through a coordinate search over λ∈{0.10,0.25,0.50}\lambda \in \{0.10, 0.25, 0.50\} and q∈{0.50,0.75,0.95}q \in \{0.50, 0.75, 0.95\}. We compare the default configuration with the configuration selected using the IEMOCAP Session-2 validation split and then fixed for five-fold retraining:

SettingMissing-VMissing-AMissing-TMean
Default77.1469.9171.0872.71
Tuned77.8568.8971.4972.74
Change+0.71−1.02+0.41+0.03

Table 6. Default versus additional validation-selected settings on IEMOCAP-4. Results report W-F1 (%); changes are tuned minus default, in percentage points.

Q4. Could the authors provide visualizations of representative success and failure cases?

We will upload the visualizations to the supplementary material(success and failure cases.pdf). Each example compares the predictions before editing, after EASE editing, and with complete input. Bars show confidence in the displayed prediction; the true label indicates whether it is correct.

Successful corrections: In the Missing-A example, EASE changes the prediction from sad to the correct label, angry. In the Missing-V example, it changes neutral to the correct label, happy. Both edited predictions agree with the complete-input model, illustrating successful correction despite the missing modality.

Failure even when complete input is correct: In another Missing-V example, both the unedited model and the complete-input model correctly predict neutral, but EASE changes the prediction to happy. This shows that an incorrect edit can damage an already correct prediction, even when the complete-input reference is reliable.

Failure when complete input is also wrong: In the second Missing-A example, the unedited prediction is correctly neutral, while both EASE and the complete-input model predict angry. Agreement with complete input therefore does not guarantee a correct answer.

These cases explain why EASE combines effect matching with ground-truth supervision, a utility gate, and magnitude control: the goal is to recover useful corrections while reducing harmful changes, rather than reproduce every reference effect indiscriminately.

Comments on the paper

3 comments

Thank you very much for your time and consideration of our rebuttal. We may be missing some context, but the references to adaptive coupling, BEMPS, and imbalance-aware mechanisms seem to concern a different method. Our paper, EASE, focuses on representation editing for missing-modality prediction. We would be very grateful if you could double-check whether these comments were intended for our submission.

Sorry for the confusion earlier. I accidentally posted a comment intended for another submission. Thanks for pointing that out.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts, and several of my original concerns have been addressed.

However, I still have reservations about the methodological contribution. The gains from effect matching remain relatively modest and inconsistent across missing-modality settings, while alternative structured carriers achieve comparable results. Although the additional backbone experiments and KD comparison are helpful, I am not fully convinced that the improvements can be primarily attributed to the proposed effect-aware mechanism rather than supervised adaptation and regularization.

Overall, the rebuttal strengthens the empirical evidence, but my main concerns remain. I will maintain my rating of 2.

Thank you for the follow-up and for acknowledging the additional evidence.

We would like to clarify that EASE combines a principled, representation-derived editing space (REP) with effect-aware learning of task-beneficial corrections. The comparable performance of alternative structured carriers suggests that the effect-aware mechanism is not restricted to REP, while REP remains a simple and effective choice for constructing the editing space.

Matching and the gate form one mechanism. CE tells the editor which label to reach, but not in which direction the missing modality would have moved the decision. Matching supplies this direction, and the gate (Eq. 21) applies it only where it helps. Without the gate, matching hurts in some settings (Tables 1.2 and 2.2); with it, EASE improves over CE + magnitude in 17 of 18 seed × condition settings across IEMOCAP-4 and CMU-MOSI (sign test p = 1.5e-4).

The gains depend on the effect itself. If each utterance is matched to another utterance's effect, with everything else unchanged, the gain disappears (−0.08 vs. CE + magnitude, p = 0.72). Likewise, the magnitude bound allows edits up to the natural size of the effect; in a linear read-out analysis on IEMOCAP-4, rescaling this limit lowers validation W-F1, and a generic L2 edit penalty without effect information falls short even at its best strength (71.89 vs. 72.73). Adaptation and regularization alone therefore do not explain the improvement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.