acceptodds
Under review as a conference paper at ICLR 2027

SenseAlign: Understanding Vision and Touch through Language at Test Time

Abstract

Tactile-visual-language models combine vision and touch through a shared language space to recognize objects and materials. In real-world deployment, changes in lighting, contact quality or sensor hardware can shift visual or tactile inputs and degrade recognition performance. When both modalities contribute to a prediction, an unreliable modality can override correct evidence from the other. Since labeled data are generally unavailable during deployment, test-time adaptation offers a way to recover recognition performance from unlabeled observations. In this setting, we find that comparing accumulated observations with the text embeddings of candidate classes reveals modality reliability differences that confidence can miss. Refining features against competing classes also improves recognition beyond geometric alignment alone. These findings motivate SenseAlign, which uses a common semantic reference for test-time modality weighting and feature correction. Alignment Trust weights each modality according to its support for shared predicted classes. Anchor Realignment uses the same class text embeddings to fit a shared feature correction that separates predicted classes from alternatives. Both modules use the same unlabeled history while keeping all encoders frozen. Extensive experiments across diverse datasets demonstrate strong recognition performance under both natural shifts and controlled corruptions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
test-time adaptation, tactile-visual-language models, multimodal fusion, tactile sensing, distribution shift, sensor reliability

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_w9FsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Tactile–visual–language models may encounter changes in modality reliability due to variations in lighting, contact quality, or sensor hardware. Less reliable modalities may even exhibit higher prediction confidence, thereby degrading fusion performance. To address this issue, this paper proposes Alignment Trust, which uses class text embeddings as semantic references to estimate the relative reliability of visual and tactile modalities and reduce the influence of unreliable modalities on classification. Furthermore, the authors introduce Anchor Realignment to learn a shared feature transformation matrix (R) by aligning fused class-level features with their corresponding text anchors and refining their class discriminability. The learned transformation is then applied to both visual and tactile features to improve class separation. Experiments on two pretrained models, UniTouch and AnyTouch, across four visual-tactile recognition benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths

(1) Practical Problem. The paper addresses a practical problem in visual-tactile recognition: unreliable modalities may remain overconfident under changing sensing conditions, thereby degrading multimodal fusion performance. By leveraging shared text embeddings as semantic references, the proposed Alignment Trust and Anchor Realignment addresses two complementary challenges: how to assign modality weights based on their reliability and how to improve the class discriminability of feature representations.

(2) Comprehensive Results. The experiments cover two pretrained models, four recognition datasets or benchmarks, and both natural distribution shifts and visual, tactile, and mixed corruptions, enabling evaluation under diverse modality reliability conditions. Furthermore, the ablation studies validate the effectiveness of both Alignment Trust and Anchor Realignment.

Critical weaknesses

(1) Novelty compared with Existing Works. One of my main concerns is that the essential differences between the proposed method and existing approaches for modality reliability estimation and feature alignment remain unclear. Alignment Trust (Eqs. (3)–(5)), which estimates modality reliability using historical statistics and reference anchors to adjust fusion weights, shares a similar idea with the Anchor-based Reliability (Eqs. (10)–(13)) and the Reliability-aware Gate Prior (Eqs. (18)–(20) and (22)) in MG-MTTA [1]. Furthermore, Anchor Realignment (Eqs. (6)–(7)), which employs orthogonal Procrustes to align modality features with text anchors, is conceptually similar to the unsupervised feature-to-text prototype alignment in U-LFA [2] (Eq. (7), with the orthogonal Procrustes formulation in Eqs. (3)–(4)). The authors should further clarify the key technical differences between these methods and the proposed approach. It would also be beneficial to provide experimental comparisons with these methods to demonstrate the superiority of the proposed designs.

[1] Chen et al. Majorization-Guided Test-Time Adaptation for Vision-Language Models under Modality-Specific Shift.

[2] Ouali et al. Black Box Few-Shot Adaptation for Vision-Language Models.

Questions for authors

Please refer to the Critical Weaknesses.

6 comments
AuthorsOpenReview ↗

W1 / Q1 — Technical differences from MG-MTTA and U-LFA

Thank you for recognizing the practical importance of unreliable modalities and the complementary roles of our two components. We are delighted that you highlighted both aspects.

SenseAlign's contribution is a shared semantic construction for online vision–touch adaptation: the same pseudo-labeled history supports both modality reliability estimation and feature correction.

MG-MTTA. Its anchors are running posterior statistics [1]. Drift between sorted current posteriors and these anchors regularizes a learned fusion gate. SenseAlign instead uses fixed class-text anchors: it groups paired observations under shared pseudo-labels, averages each modality's standardized support for its assigned anchor across historical groups, and applies softmax to obtain fusion weights directly. This compares vision and touch against the same class-specific reference when their reliability differs.

U-LFA. It alternates sample-to-class assignment and feature-map fitting [2]. SenseAlign builds shared historical class groups from raw equal fusion: separate visual and tactile statistics estimate reliability, while fused class means fit one constrained correction applied to both sensors. Both operations use prior history, which is updated after each batch's predictions. Procrustes itself is an established mathematical tool; our contribution lies in how the shared semantic history organizes these complementary operations for vision and touch.

We will make these distinctions explicit. Direct comparisons are reported in our response “W1 / Q1 — Direct comparisons with MG-MTTA and U-LFA”.

References

[1] Chen et al. Majorization-Guided Test-Time Adaptation for Vision-Language Models under Modality-Specific Shift. ACM MM 2026. Official released configuration.

[2] Ouali et al. Black Box Few-Shot Adaptation for Vision-Language Models. ICCV 2023.

AuthorsOpenReview ↗

W1 / Q1 — Direct comparisons with MG-MTTA and U-LFA

We evaluated both methods on AnyTouch and UniTouch, keeping their protocols separate. Accuracy (%) averages backbones equally; TAG-C averages its three settings, and Average weights the three dataset families equally.

MG-MTTA: online adaptation. Our vision–touch implementation of MG-MTTA follows the official released code [1], updating encoder normalization parameters and the fusion gate. Both methods receive identical streams at batch size 16 and seed 0; MG-MTTA's released batch size is 64. MG-MTTA adapts before prediction, while SenseAlign updates afterward.

MethodOFRRCT MaterialTAG-CAverage
MG-MTTA46.5945.9633.0341.86
SenseAlign62.2955.5560.5759.47

SenseAlign leads in nine of ten backbone–setting combinations; MG-MTTA leads on UniTouch RCT Material (71.82% vs. 64.95%). The low TAG-C scores include single-class collapse on AnyTouch Visual and Mixed. Freezing the encoders while retaining gate and anchor updates prevents this collapse, implicating encoder drift in the tested configuration. SenseAlign's frozen encoders avoid this failure path.

U-LFA: support fitting, then frozen evaluation. Both methods fit the same unlabeled support set and remain frozen on disjoint query groups. U-LFA [2] fits vision and touch independently, and we average their logits. This gives both methods the same support information and no query-time adaptation.

MethodOFRRCT MaterialTAG-CAverage
U-LFA41.3526.2643.6637.09
SenseAlign63.5252.2557.7957.85

SenseAlign leads in all ten backbone–setting combinations averaged over five seeds. Weak-sensor assignments may distort U-LFA's independently fitted map, whose scores still contribute equally to fusion. Its uniform-class assignment prior may also mismatch support sets that are imbalanced or omit classes. We will clarify the implementations and protocols alongside the results.

References

[1] Chen et al. Majorization-Guided Test-Time Adaptation for Vision-Language Models under Modality-Specific Shift. ACM MM 2026. Official released configuration.

[2] Ouali et al. Black Box Few-Shot Adaptation for Vision-Language Models. ICCV 2023.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer,

Following your suggestions, we have clarified the technical differences from MG-MTTA and U-LFA and added direct experimental comparisons with both methods on AnyTouch and UniTouch. When you have a chance, we would appreciate your feedback on these clarifications and results. We would be happy to discuss any remaining concerns.

Thank you again for your time and consideration.

Reviewer_CPw1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SenseAlign is a test-time adaptation method for tactile-visual-language models, which keeps all encoders frozen and adapts only from the unlabeled test stream. It accumulates pseudo-labeled class means for each modality, and uses the class text embeddings to weight vision against touch and to fit a feature correction that separates the predicted classes from the other candidates. On the UniTouch and AnyTouch models, the method outperforms equal fusion and existing test-time adaptation baselines on object and material recognition.

Critical strengths
  1. Lightweight and practical adaptation: SenseAlign keeps all encoders frozen and does not require any labels or gradient updates. This makes it computationally cheap to run on released tactile-visual-language checkpoints during deployment. The method improves over equal fusion on every evaluated task for both backbones, and still outperforms the strongest baseline when a shared configuration is used across all scenarios.

  2. Extensive evaluation experiments: The experiments cover two backbones, natural and controlled shifts, and ten test-time adaptation baselines. The paper also compares its weighting rule against confidence, entropy and margin-based rules under the same features and memory, and reports the cases where the method loses. This makes it possible to see how much each component contributes to the final performance.

Critical weaknesses

The comparison with baselines is not fully balanced: The main table results of SenseAlign use hyperparameters selected separately for each scenario using the test labels, while all baselines use one shared configuration without tuning. Several baselines collapse under this setting, which might be suggesting that they are not well configured for these backbones. The sensitivity study also shows that SenseAlign's accuracy can change by a large margin across settings. Although the paper reports a single shared configuration that still outperforms TDA, the gaps in the main tables may be unfair.

Questions for authors

Would it be possible to select the hyperparams of SenseAlign on a held-out stream, and tune the learning rate of the gradient-based baselines in the same way? If SenseAlign still outperforms the baselines under this protocol, it would support the claim that the improvement does not come from per-scenario tuning.

4 comments
AuthorsOpenReview ↗

W1 / Q1 — Comparable tuning on separate development streams

Thank you for recognizing SenseAlign's practical, lightweight design and the breadth of our evaluation. We are delighted that you highlighted its gains with a shared configuration.

We used a shared learning rate of 10⁻⁴ for gradient-based baselines to control a known source of variation in TTA evaluation [1]. We agree that the original comparison gave SenseAlign more opportunities for configuration selection.

To provide comparable tuning opportunities, we selected settings on separate labeled development streams: VITA for RCT, and RCT-dev for the other six scenarios. We evaluated six prespecified configurations each for SenseAlign (β and g), TDA (cache capacities and correction strength), OGA (cache capacity and τ), and StatA (pseudo-label and Laplacian regularization weights), alongside six learning rates for participating gradient-based baselines.

For each method, backbone, and development source, we selected the configuration with the highest mean accuracy across the same three development streams, using material-macro accuracy on RCT-dev and object-macro accuracy on VITA. Each selected configuration was then held fixed across its corresponding target scenarios when re-evaluating the previously tested streams. Online adaptation used only unlabeled target inputs, without access to the original source training data.

Setting / methodSenseAlignStatATDADeYOSuMi FP32SARTENT-LNOGARoTTATSAREAD
Fixed (%)56.6247.5649.0048.6942.0647.4838.3541.7940.3635.6411.01
Dev-selected (%)55.9749.1748.6748.0347.3847.0146.8446.2444.6639.6812.12
Change (pp)−0.65+1.62−0.33−0.66+5.32−0.47+8.50+4.45+4.30+4.04+1.11

Accuracy (%) averages all seven scenarios on both backbones equally (14 backbone–scenario pairs). Fixed uses the archived baseline settings and the shared SenseAlign configuration; changes use unrounded values.

With development-set selection, SenseAlign achieves 55.97%, compared with 49.17% for the strongest baseline, StatA, and 48.03% for the strongest gradient-based baseline, DeYO (+7.94 pp). Its advantage remains with settings selected on separate development data.

[1] Zhao et al. On Pitfalls of Test-Time Adaptation. ICML 2023, §4.1 and Appendix A.

AuthorsOpenReview ↗

W1 — Baseline adaptations and numerical stability

AnyTouch and UniTouch are based on CLIP and support vision, touch, and language. Few evaluated algorithms directly support this combination, so the baseline results use our adaptations to these backbones.

For READ, we apply its official loss to averaged visual and tactile logits and update the available QKV parameters, without its original fusion-attention architecture. TDA, OGA, and StatA run separately on vision and touch before their outputs are averaged. SuMi replaces audio with touch and uses one pooled token per modality.

To address numerical failures in the mixed-precision AnyTouch SuMi runs on the three TAG-C scenarios, we reran SuMi in FP32. The SuMi FP32 column in our response “W1 / Q1 — Comparable tuning on separate development streams” reports completed FP32 evaluations for both fixed and development-selected settings. We will explicitly label the baselines as our adaptations in the revised tables and document their architectural mappings and numerical precision.

AuthorsOpenReview ↗

W1 — Interpreting the wide hyperparameter sweep

Our sensitivity study deliberately spans extreme settings: β from 0 to 1 moves from no feature correction to full correction, while g from 0 to 64 moves from equal fusion to strongly favoring one modality. These settings substantially change the prediction rule, so the full-range accuracy spread alone does not establish sensitivity to small parameter changes.

We will clarify the sweep's purpose and interpretation. The development-set comparison in our response “W1 / Q1 — Comparable tuning on separate development streams” shows that SenseAlign retains its advantage with settings selected on separate development streams and fixed before evaluation.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer,

Following your suggestions, we have added comparisons with hyperparameters selected on separate development streams and clarified the baseline adaptations and interpretation of the sensitivity study. We would appreciate your thoughts on whether these additions address your concerns about the comparison protocol. Please let us know if any points need further clarification.

Thank you again for your time and consideration.

Reviewer_mHGoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SenseAlign addresses unlabeled test-time adaptation for pretrained vision–touch–language models under distribution shift. It uses predictions from the original equal-weight fusion to construct a shared history of unlabeled observations. Alignment Trust adjusts fusion weights according to each modality’s semantic support for shared pseudo-label assignments. In parallel, Anchor Realignment learns a shared feature correction from class-wise fused feature means and fixed text anchors, then applies it to both modalities to improve class discrimination. All encoders remain frozen. Experiments evaluate UniTouch and AnyTouch under natural distribution shifts and controlled corruptions and show convincing improvements.

Critical strengths
  • The problem is clearly motivated, and the evaluation covers relevant modality relationships. The experiments include settings where touch is unreliable, touch is dominant, and both modalities provide useful information. These settings demonstrate the limitations of fixed fusion weights and support the need to adapt modality contributions. The results suggest that combining semantic reweighting with feature realignment can mitigate unfavorable modality contributions and improve recognition beyond the original equal-weight fusion.

  • The controlled comparisons and mechanism analyses provide meaningful evidence for the proposed components. The results also show that moving features closer to text anchors is not enough; classification refinement further improves class separation. The ablations support combining modality weighting with feature correction.

Critical weaknesses
  • Figures 2 and 3 contain useful information but are difficult to interpret without additional explanation. For example, Figure 3(b) should define how accuracy, confidence, and alignment are computed. The relationship between confidence and accuracy discussed in the text is not immediately apparent from the visualization. Some comparisons appear to show confidence and accuracy moving in the same direction.

  • The reliability of the shared pseudo-label memory remains insufficiently characterized. Alignment Trust measures support for shared pseudo-label assignments, which may differ from support for the true classes. If the original equal-weight fusion makes systematic semantic errors, can the proposed method recover, or could both components reinforce those assignments? The paper would benefit from reporting pseudo-label class distributions alongside ground-truth distributions, class coverage, and the purity or accuracy of accepted memory groups across datasets. Controlled pseudo-label noise experiments would further clarify the method’s failure conditions.

  • The cumulative memory may respond slowly to changes within a test stream. Since μc,bm\mu_{c,b}^m is indexed by batch, it aggregates all previously accepted observations without temporal decay or removal. As the accumulated weight grows, a new batch generally contributes less to the class means. This may become problematic when modality reliability changes abruptly after a period. Can the authors consider some sliding window or exponentially decayed strategies to clarify the trade-off between historical stability and responsiveness.

Questions for authors
  • Figure 4 could better explain the connection between Alignment Trust and Anchor Realignment. The current figure arrangement does not clearly distinguish the shared pseudo-label grouping, the modality statistics used for weighting, and the fusion correction process. Since the two components share a semantic reference rather than jointly optimizing each other, a clearer depiction of their information flow and complementary roles would improve understanding.

  • The author can consider improving the problem definition. The proposed method seems to be an improvement plug-in building upon a pre-trained vision-touch-language alignment space. What's its connect with test time adaptation?

7 comments
AuthorsOpenReview ↗

W1 — Interpreting accuracy, confidence, and alignment in Figures 2–3

Thank you for recognizing the clear motivation of our work and the useful information in Figures 2 and 3. We are delighted that you found our mechanism analyses meaningful.

We apologize that the captions and text did not explain the metrics and comparisons clearly.

Figure 2 illustrates how the relative contributions of vision and touch change across sensing conditions. Figure 2(a) reports the fraction of each class on the correct side of the displayed two-class boundary; Fig. 2(b) reports full-stream accuracy and gains over frozen equal fusion. In Fig. 3(b), accuracy is frozen-branch top-1 accuracy, confidence is mean maximum softmax probability, and alignment averages standardized assigned-anchor scores across nonempty historical groups at the end of the stream.

The intended comparison is between modalities within each setting. Confidence and accuracy both favor vision for UniTouch. For AnyTouch, however, confidence favors vision while accuracy and alignment favor touch. This illustrates that confidence can misrank modality reliability, even though it agrees with accuracy in other settings. We will clarify the captions and interpretation and add the metric formulas and averaging conventions to the revised manuscript.

AuthorsOpenReview ↗

W2 — Shared-history robustness, class coverage, and purity

SenseAlign constructs memory from the original equal-weight fusion, so adapted predictions never become future assignments. The encoders remain frozen and realignment applies a constrained, partial correction. We tested the complete method while both modules continued updating from corrupted shared history.

Controlled noise tests. We randomly replaced accepted pseudo-labels with uniformly sampled alternative classes, on top of natural assignment errors. We average native stream scores within each dataset (TAG-C Mixed, 48 TacQuad streams, five OFR splits, and RCT material recognition), then weight the four datasets and two backbones equally.

Additional replacement rate0%10%20%40%70%100%
SenseAlign accuracy (%)55.4154.1253.2052.2949.7332.06
Δ vs. best baseline (pp)+8.91+7.61+6.70+5.79+3.23−14.44

The reference is the best TTA baseline reported in the paper, without injected noise, aggregated using the same averaging scheme (46.50%). At 70% additional replacement, SenseAlign retains 49.73% accuracy, exceeding this reference by 3.23 percentage points. Accuracy falls to 32.06% at 100%, showing a limit to this robustness. Replacement rates measure changes to raw pseudo-labels, not their error rates against ground truth.

We also persistently reassigned accepted Brick labels to Rock in both histories on UniTouch TAG-C Mixed. This intervention changes the labels written into memory while keeping the input stream and original equal-fusion predictions unchanged. Across 2,048 injection batches (3,325 rewritten labels), accuracy was 55.02% versus 55.00% for the matched control, and Brick recall was 94.18% versus 94.27%.

Across all 32 consecutive windows of 1,024 observations during injection, accuracy and Brick-recall drops relative to the matched control never exceeded 0.49 and 1.43 percentage points, respectively. These results support robustness to persistent Brick-to-Rock errors written into shared history.

Class coverage and purity. We audited shared memory across four datasets and both backbones. Correct class coverage is the fraction of classes present in the stream with at least one correctly assigned memory write. Assignment purity is the fraction of accepted writes whose pseudo-label matches ground truth, while margin-weighted purity is the total margin weight of those correct writes divided by the total margin weight of all accepted writes. Ground-truth labels are used only for offline evaluation.

Dataset / conditionCorrect class coverageAssignment purityMargin-weighted purity
TAG-C, all three conditions100.0053.4377.02
TacQuad, clean50.00–62.5044.3765.49
TacQuad, visual corruptions43.48–70.4538.4758.24
ObjectFolder Real83.33–100.0057.8475.46
RCT78.5739.6646.90

All entries are percentages, averaged equally over AnyTouch and UniTouch. Coverage ranges span streams after backbone averaging; purity is pooled within each backbone before averaging. The TAG-C row includes all three conditions.

Normal history improves mean accuracy from 46.54% without adaptation to 55.41% across four datasets and both backbones. This gain is obtained despite natural assignment errors and class imbalance in memory. For example, on RCT, paper cardboard makes up 33.93% of observations and 5.59% of accepted pseudo-labels, averaged across both backbones.

Margin-weighted purity is higher than assignment purity in every reported condition, consistent with more reliable entries contributing more strongly to the historical statistics. Margin weighting reduces low-margin contributions, while partial realignment retains part of the original representation. The accuracy gains and controlled noise tests show that SenseAlign uses imperfect historical information effectively while retaining gains under substantial additional pseudo-label corruption. We will add the class-distribution analysis and metric definitions to the revised manuscript.

AuthorsOpenReview ↗

W3 — Cumulative history versus EMA and sliding windows

Cumulative history retains evidence for classes absent from recent batches, giving both modules a reference spanning the observed stream. New observations have less influence as class mass grows, so this stability can come at the cost of slower adaptation after an abrupt change.

We compared cumulative history, EMA with a 256-observation half-life, and a 256-observation sliding window on identical UniTouch streams. Cumulative history has the highest accuracy in the sustained-fault comparisons, including a 2.02 pp advantage over the window in the last 512 visual-fault observations.

Sustained faultCumulativeEMASliding window
Visual, all 2,048 observations56.28%56.12%55.50%
Visual, last 512 observations57.81%57.36%55.79%
Tactile, all 2,048 observations59.96%59.81%58.72%
Tactile, last 512 observations61.85%61.59%61.78%

After repair, EMA achieves higher accuracy over the next 1,024 observations. Cumulative history remains above the window following tactile repair. Both tables average three orders of the same source pool.

Recovery stage: next 1,024 observationsCumulativeEMASliding window
Visual fault → repair60.32%62.04%60.90%
Tactile fault → repair62.37%62.89%60.97%

We also expanded the temporary-fault comparison to both backbones on TAG-C, TacQuad, and OFR, using 6,448 distinct source frames. TAG-C uses 1,536 clean observations followed by three cycles of 512 fault and 2,048 clean observations, testing both fault directions and reusing frames in later cycles. TacQuad adds visual-corruption tests (128 fault / 256 clean observations); OFR adds tactile-crop tests (160 / 512).

PhaseCumulativeEMA256Window256
During faults58.58 (+1.63)58.79 (+1.57)58.40 (+1.52)
After restoration60.70 (−0.49)60.49 (−0.40)59.79 (−0.39)

Entries report accuracy (%) and deficit from each policy's fault-free replay of the same frames in the same order (pp, parentheses; negative values indicate higher accuracy). We average backbones and tested conditions within each dataset, then weight the three datasets equally.

The policies have similar mean fault accuracy and small aggregate deficits after temporary faults. Cumulative history retains the highest mean post-restoration accuracy, while the long-fault diagnostic shows that EMA can respond better after abrupt repair. We will clarify this stability–responsiveness trade-off in the revised manuscript.

Reviewer_maU1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The robot is tasked to identify the object from a given list of object names, provided it has access to touch and vision input. A word is selected if its representation is most similar overall to the vision and touch inputs. Both vision and touch input get to give a vote on which object it is. However, in the real world, either vision or touch sensor can be unreliable hence making the prediction unreliable. The paper proposes two fixes. 1) Alignment trust: Initially the robot places trust on both vision and touch sensor equally and keeps on making the prediction. It checks on which sensor is more consistent with naive 50/50 feature fusion’s predictions compared with the other object names. A less consistent sensor gets a low weighing value for the final prediction. 2) Anchor Realignment: The paper also proposes using feature rotation so each object's representation is more aligned with its predicted word. The proposed work makes these adjustments on the fly to give better prediction of the object than naive baseline which equally weighs both sensors.

Critical strengths

Neither retraining nor label is needed and the method works with off-the-shelf models. The paper provides an interesting insight: the confident sense can be less accurate in this specific setup. The method doesn't bias towards one sensor as in: it sometimes prefers vision, sometimes touch input depending on their reliability.

Critical weaknesses
  • The work heavily relies on its own 50/50 guesses. Both proposed components of SenseAlign build on its own guesses / state. For example, alignment trust judges each sense from how its features support the robot's own naive 50/50 guesses. If many naive guesses already happen to be wrong, its adjustment may be unreliable. Also, anchor realignment rotates the features of the both sensors’ representations to match the guessed word's representation. If 50/50 prediction was wrong, it pulls the objects to the wrong word.
  • The weighing factors are decided on the batch basis. Each input may have a different case: in one case, both sensors may be fine while in another case: just one of the sensors could be faulty. Batch-wise adjustment seems somehow suboptimal.
  • The proposed system may be slow to adapt with sudden changes. What if suddenly a sensor is fixed and is no longer faulty? The weighing factors are updated every batch but they are based on past observations. Suddenly fixed faulty sensors won't be reflected on time.
  • Some good part of the gain can come simply from ignoring the input of one of the faulty sensors.
  • The method doesn't exhibit the best empirical performance everywhere. The paper is not very easy to follow. The presentation could be improved.
Questions for authors

None

4 comments
AuthorsOpenReview ↗

W1 — Robustness to incorrect pseudo-labels in shared history

Thank you for recognizing that SenseAlign works with off-the-shelf models without retraining or labels. We are delighted that you found the confidence–accuracy insight interesting.

SenseAlign keeps the encoders and text anchors frozen and fits one shared orthogonal transform from class-wise history. Groups with dispersed features contribute less during discriminative refinement.

The correction is partial (β=0.2 here), preserving much of the original representation. Future pseudo-labels come from the original equal-weight predictions, so adapted outputs never feed back into memory assignments.

We tested full SenseAlign by independently replacing each admitted pseudo-label with a uniformly sampled different class with probability p. Both modules updated online from these shared assignments, with admission decisions and margin weights unchanged. This adds noise to naturally imperfect pseudo-labels; 0% does not mean error-free labels.

We average native stream scores within each dataset (TAG-C Mixed, 48 TacQuad streams, five OFR splits, and RCT material recognition), then weight the four datasets and two backbones equally.

Replacement probabilitySenseAlign accuracy (%)Δ vs. best baseline (pp)
0%55.406+8.905
10%54.116+7.615
20%53.203+6.702
40%52.294+5.793
70%49.732+3.231
100%32.057−14.445

The reference is the best TTA baseline reported in the paper, without injected noise, aggregated using the same four-dataset, two-backbone averaging scheme (46.501%). Even with 70% additional random pseudo-label replacement, SenseAlign achieves 49.732%, exceeding this reference by 3.231 percentage points.

AuthorsOpenReview ↗

W2–W3 — Historical weighting and response to sensor repair

W2 — Historical versus instance-wise weighting.

Class-wise historical aggregation reduces dependence on individual corrupted inputs, while shared semantic assignments provide a common reference for comparing the two modalities.

To isolate fusion weighting, we fixed history, Anchor Realignment, and corrected branch scores, then changed only the fusion rule. Every rule, including equal fusion, retains Anchor Realignment.

Instance-wise semantic weights use the current query’s original equal-fusion pseudo-label: we standardize each modality’s uncorrected feature–text similarities across classes, take the standardized score of that shared label, and apply a softmax across the two modalities with the same inverse temperature g=4 as historical weighting.

DatasetHistorical weightsInstance-wise semanticEqualInstance-wise margin
TAG-C61.77 (2.43)59.21 (3.64)60.73 (3.10)59.89 (2.64)
TacQuad45.38 (6.01)42.80 (7.28)40.25 (3.88)42.81 (5.24)

Corrupted-query accuracy (%) and clean-minus-corrupted drops (pp, parentheses) are computed from paired inputs sharing the same past state. We use TAG-C’s 15 visual and 7 tactile corruption conditions and TacQuad’s 15 visual corruptions across three sensors, evaluated after history initialization. We average source groups within conditions, then conditions and backbones equally; differences use unrounded values.

Historical weighting gives the highest mean accuracy on both datasets. Compared with instance-wise semantic weighting, accuracy improves by 2.56 pp on TAG-C and 2.57 pp on TacQuad; corruption-induced drops shrink by 1.21 and 1.27 pp, respectively.

W3 — Response to sensor repair.

TTA settings range from stationary target distributions to continual shifts, which may be gradual or abrupt [1], and mixed-domain streams [2]. Sudden sensor repair represents an abrupt change in modality reliability within a continuing stream. Balancing rapid responses with stable adaptation remains a challenge in dynamic TTA [3].

SenseAlign uses cumulative class-wise evidence to stabilize modality reliability estimates under noisy observations. To examine its response to sudden changes, we compare cumulative history with EMA and sliding windows on identical fault-and-recovery streams.

In the UniTouch long-fault diagnostic, EMA recovers sooner than cumulative history in two of three visual-repair orders and ties in the third; both reach the earliest confirmable recovery after tactile repair. Recovery in that diagnostic requires two consecutive 256-observation windows no more than 2 pp below a reference that resets history at repair.

We expanded the temporary-fault comparison to both backbones on TAG-C, TacQuad, and OFR, using 6,448 distinct source frames. TAG-C tests both fault directions: 1,536 clean observations followed by three cycles of 512 fault and 2,048 clean observations, with frames reused in later cycles. TacQuad adds visual-corruption tests (128 fault / 256 clean observations), and OFR adds tactile-crop tests (160 / 512).

PhaseCumulativeEMA256Window256
During faults58.58 (+1.63)58.79 (+1.57)58.40 (+1.52)
After restoration60.70 (−0.49)60.49 (−0.40)59.79 (−0.39)

Entries report accuracy (%) and deficit from the same policy's matched fault-free replay of the same frames in the same order (pp, parentheses; negative values indicate higher accuracy). We average both backbones and tested conditions within each dataset, then weight the three datasets equally.

The policies have similar mean fault accuracy and small aggregate deficits after restoration; cumulative history has the highest mean post-restoration accuracy. In the longer TAG-C streams, every 256-observation recovery window under cumulative history stays no more than 2 pp below its matched fault-free reference, supporting stability after temporary faults.

References

[1] Wang et al. Continual Test-Time Domain Adaptation. CVPR 2022.

[2] Niu et al. Towards Stable Test-Time Adaptation in Dynamic Wild World. ICLR 2023.

[3] Yuan et al. Robust Test-Time Adaptation in Dynamic Scenarios. CVPR 2023.

AuthorsOpenReview ↗

W4–W5 — Contributions of the downweighted modality and condition-level results

W4 — Contribution of the downweighted modality.

Our controls show additional benefits from multimodal history and from retaining the downweighted branch in fusion.

On identical TAG-C streams with fixed settings, C adapts and predicts using vision alone from the first observation; B uses full SenseAlign’s multimodal history and realignment but predicts from corrected vision alone; A is full SenseAlign.

Accuracy (%) across all six backbone–condition combinations; the mean weights them equally. Contrasts are in pp and computed before rounding.

Backbone / conditionC: Vision-only history / visionB: Multimodal history / visionA: Full SenseAlignHistory B−C (pp)Total A−C (pp)
AnyTouch / Visual fault55.8756.5762.41+0.70+6.54
AnyTouch / Tactile fault68.6069.0965.99+0.49−2.61
AnyTouch / Mixed58.4959.2661.45+0.77+2.96
UniTouch / Visual fault50.1350.7756.88+0.64+6.76
UniTouch / Tactile fault58.7659.9660.81+1.20+2.05
UniTouch / Mixed51.2752.1755.10+0.91+3.83
Six-panel mean57.1957.9760.44+0.79+3.25

Conditional 95% intervals are [1.74, 4.67] pp for the mean A−C contrast and [0.09, 1.40] pp for the mean B−C contrast. B−C isolates the historical contribution through visual realignment; it does not capture all effects on Trust. The AnyTouch tactile-fault exception is shown above.

To isolate the currently downweighted branch, we removed only the strictly lower-weight branch while fixing history, correction, and original weights. We exclude equal-weight batches and average eligible-stream accuracies equally. Conditional on completed histories, intervals use 5,000 paired bootstrap draws over 117 capture groups. This estimator differs from A/B/C, so the gains are not additive.

Readout on strict non-tied subsetAccuracy (%)Gain over higher-weight only (pp; 95% CI)
Full SenseAlign60.44+6.25 [5.15, 7.27]
Higher-weight branch only54.19Reference

Soft-score fusion was also correct on 6,251 presentations (1,550 source pairs; 97 captures) where both corrected branches’ individual predictions were wrong. These outputs cannot be obtained by selecting either branch’s top-1 label.

W5 — Condition-level results and clarity.

Regarding condition-level results, SenseAlign has the highest mean accuracy for each backbone in both main tables. Exceptions include TDA on AnyTouch TacQuad clean (46.3% vs. 45.5%) and SAR on UniTouch TAG-C Visual (58.9% vs. 57.7%) and Mixed (57.7% vs. 57.1%).

These gaps may reflect TDA’s sample-level cache retaining local structure and SAR’s ability to update encoder normalization. SenseAlign uses class-wise history and frozen representations.

We apologize for the unclear presentation. We will clarify the condition-dependent results, define the quantities in Figures 2–3, and revise Figure 4 and its caption to explain the information flow between the modules.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer,

We have supplemented our responses with experiments on pseudo-label robustness, historical versus instance-wise weighting, sensor fault and recovery, and the contribution of the downweighted modality. When you have a chance, we would appreciate your feedback on whether these results address your concerns. We would be happy to clarify any remaining points.

Thank you again for your time and consideration.

Comments on the paper

4 comments

Dear Reviewers,

We are currently in the discussion phase. The authors have responded to the comments, and I would greatly appreciate it if you could take a moment to check the rebuttal and leave any follow-up comments or questions. If the response has addressed your concerns, please also consider updating your review accordingly.

Your feedback is very valuable to both the authors and the decision process. Thank you again for your help.

Best regards, AC

AuthorsOpenReview ↗

General Response and Summary of Additional Evidence (1/2)

We thank the reviewers and Area Chair for the constructive discussion and recognition of SenseAlign's practical motivation and lightweight design. Below, we summarize the main concerns, the experiments addressing them, and the conclusions supported by those experiments.

Accuracy is reported in percent. Gains in percentage points (pp) are computed before rounding.

1. Novelty — w9Fs

SenseAlign introduces a shared semantic history for online vision–touch–language adaptation. The same class groups support both sensor reliability estimation and a joint feature correction, with text providing a common reference and all encoders remaining frozen.

  • MG-MTTA [1] uses posterior statistics to guide a learned fusion gate. SenseAlign instead derives weights directly from each sensor's historical support for the same assigned text class.
  • U-LFA [2] alternates individual support-sample assignments and map fitting. SenseAlign maintains groups defined by original equal-fusion predictions and fits one constrained correction for both sensors from fused historical class means.

This design allows paired sensor observations to contribute to both reliability estimation and future feature correction. The modality-contribution controls in point 4 of this response test whether touch supplies useful information to current predictions and subsequent visual adaptation.

2. Comparison fairness — CPw1; w9Fs's batch-size follow-up

Development-set selection (CPw1). To test whether the advantage survives comparable tuning opportunities, we evaluated six prespecified candidates per participating method, including six learning rates for gradient-based baselines. Settings were selected on separate labeled development streams and then held fixed across their corresponding target scenarios. Online adaptation used only unlabeled target inputs, without source training data. Averaging AnyTouch and UniTouch equally across OFR, RCT, TacQuad clean/corrupted, and TAG-C visual/tactile/Mixed scenarios, SenseAlign achieves 55.97% versus 49.17% for the strongest evaluated baseline, StatA. The advantage therefore persists when both SenseAlign and its competitors receive development-based selection. Selection protocol and full results.

Batch size (w9Fs). To address whether batch size 16 disadvantaged MG-MTTA, we reran both methods at its released batch size of 64, using the same checkpoints, class prompts, and input order for each backbone. Both processed the full streams online, and we scored identical, prespecified query examples. Averaging the same seven scenarios and two backbones equally at seed 0, SenseAlign achieves 55.38% versus 46.42% (+8.96 pp). This directly confirms that the observed advantage persists at MG-MTTA's released batch size. B64 protocol and comparison table.

3. Pseudo-label robustness — maU1; mHGo

The concern is that errors in the shared history could mislead both modules. We therefore corrupted the labels written to that history after prediction: each admitted pseudo-label was replaced by a uniformly sampled different class with probability 70%. Both modules received these modified assignments, while admission decisions and margin weights remained unchanged. The intervention adds noise to the backbone's naturally imperfect predictions.

Across TAG-C Mixed, TacQuad, OFR, and RCT, weighting datasets and backbones equally, SenseAlign achieves 49.73%, compared with 55.41% without injected noise and 46.50% for the strongest baseline reported in the paper without injection. Thus, even substantial additional corruption of the shared assignments leaves a 3.23 pp advantage over that unperturbed baseline. This supports robustness to random memory-label corruption, consistent with frozen encoders, partial feature correction, and history labels that do not depend on adapted predictions. Noise-injection protocol and full results.

AuthorsOpenReview ↗

General Response and Summary of Additional Evidence (2/2)

4. Contribution of the downweighted modality — maU1

We tested whether reducing a sensor's influence still preserves useful information through two distinct controls on TAG-C's visual-fault, tactile-fault, and Mixed streams, using both backbones. Each mean weights the six backbone–scenario combinations equally.

Current prediction. We held history, feature correction, and the original fusion weights fixed, then removed only the lower-weight branch on batches with unequal weights. Mean accuracy falls from 60.44% to 54.19%. Because the historical state is identical and only the current branch's contribution is removed, the 6.25 pp difference demonstrates useful predictive information in the downweighted branch.

Historical adaptation. Both variants made their final predictions using only vision. One adapted from the start using vision alone, while the other also used past paired tactile observations to build its history and feature correction. On the same streams, mean visual accuracy rises from 57.19% to 57.97% (+0.79 pp), with gains in all six backbone–scenario combinations. Since current tactile scores are excluded from both final predictions, this improvement measures the benefit of including touch in adaptation history. Paired controls and full results.

5. Stability and responsiveness — maU1; mHGo

The concern is whether accumulated history becomes unhelpful when sensor reliability changes. We compared cumulative history, EMA, and sliding windows on the original TAG-C Mixed stream, which interleaves clean, visual-corruption, tactile-corruption, and joint-corruption observations. These recurring conditions test whether earlier class evidence remains useful despite frequent reliability changes.

We kept features, prompts, stream order, and other settings fixed. EMA's half-life and the window length were 16 native batches, nominally 256 observations. On the complete stream of 94,898 sample presentations per backbone, cumulative history achieves 58.27%, compared with 57.75% for EMA and 56.54% for sliding windows, averaging both backbones. This supports retaining accumulated evidence in these recurring-condition streams. Mixed-stream protocol and full results.

We separately tested repair on UniTouch using 2,048 visual-fault observations followed by 1,024 restored observations, without resetting history. Averaged over three stream orders, accuracy during restoration is 60.32% for cumulative history and 62.04% for EMA. This shows where forgetting can help after repair and makes the stability–responsiveness trade-off explicit. Fault–repair protocol and results.

6. Method clarity — mHGo; maU1; CPw1

We clarified how shared pseudo-labels are formed, which historical statistics each module reads, and how corrected scores are fused. We will revise Figures 2–4 and their captions, define the reported metrics explicitly, and document the baseline adaptations to the vision–touch–language backbones.

References

[1] Chen et al. Majorization-Guided Test-Time Adaptation for Vision-Language Models under Modality-Specific Shift. ACM Multimedia, 2026.

[2] Ouali et al. Black Box Few-Shot Adaptation for Vision-Language Models. ICCV, 2023.

Dear Reviewers,

Thank you again for your time and thoughtful reviews. We would greatly appreciate your further feedback on our responses and additional results. If any questions or concerns remain, please let us know! we would be so happy to discuss them. We highly value your feedback and recognition of our work.

Best, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.