acceptodds
Withdrawn from ICLR 2027

Seeing Heat, Acting by Touch: Thermally Grounded Prompting for Vision-Tactile-Language-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Tactile-Language-Action (VTLA) models augment Vision-Language-Action (VLA) policies with tactile feedback, enabling robots to respond to contact deformation, slip, and grasp stability during physical interaction. However, tactile sensing is available only after contact and cannot safely identify pre-contact target properties such as object temperature or electrical energization. A direct solution is to inject both thermal and tactile tokens into a pretrained action backbone. With limited robot demonstrations, however, jointly learning heterogeneous representations, cross-modal alignment, and continuous control from action supervision is data-inefficient and prone to RGB-dominated modality imbalance, potentially suppressing thermal and contact-dependent tactile cues. Scaling synchronized RGB–thermal–tactile–language–action data to alleviate this problem is also substantially more resource-intensive than collecting conventional RGB-based robot demonstrations. To address these challenges, we propose Thermo-VTLA, a hierarchical framework that separates thermal physical-state grounding from contact-rich VTLA control. A thermally grounded vision-language model receives paired RGB–thermal observations and the original instruction, and produces a constrained executable prompt. A prompt-conditioned VTLA controller then executes the command using RGB observations, robot state, and visuotactile feedback, without receiving thermal tokens. To effectively incorporate tactile feedback without disrupting the pretrained vision-language representation, we introduce a two-stage contact-aware tactile integration strategy. Contact-aware gating first determines tactile validity from the interaction state, after which gated fusion adaptively regulates the contribution of valid tactile representations to the action context. This decomposition preserves tactile sensing in the high-frequency control loop while moving thermal reasoning to an explicit semantic interface. We evaluate Thermo-VTLA on beverage retrieval, bread manipulation, and electrical-plug disconnection.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Tactile-Language-Action Models, Visuotactile Manipulation, Multimodal Robot Learning, Thermal Perception

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5ToDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Thermo-VTLA separates pre-contact thermal target grounding from vision–tactile control. A frozen VLM selects a task-specific spatial command from RGB–thermal observations, and a SmolVLA-based controller executes it using RGB, robot state, and tactile feedback.

Critical strengths
  1. Well-motivated modular design. Thermal sensing supports pre-contact target selection, while tactile sensing supports contact-rich execution through an inspectable command interface. This separation provides a practical way to distinguish grounding failures from execution failures and avoids requiring the low-level policy to learn thermal–action alignment from robot demonstrations (Sections 3.1–3.4).

  2. Meaningful real-robot evidence and a useful ablation design. The reported improvement over joint thermal–tactile token injection is substantial in the evaluated settings: 93.3% versus 45.0% average success (Table 2). The tactile ablations use ground-truth executable prompts, appropriately isolating execution from grounding errors; removing tactile feedback reduces success from 93.3% to 73.3% (Section 4.4, Table 3).

Critical weaknesses
  1. The contact-gating definition requires reconciliation. Equation (7) measures consecutive-frame marker motion, whereas Appendix D.1 describes displacement from reference positions. A stable grasp can maintain deformation with little consecutive-frame motion, so the stated criterion need not indicate sustained contact. The actual reference, thresholding, and any state-retention mechanism must be clarified before this central contribution can be assessed. This may be a reporting error rather than an implementation failure (pp. 5–6 and 17).

  2. The comparisons do not sufficiently isolate the hierarchical design's contribution. The proposed system uses a frozen GPT-5.6-sol reasoner to supply resolved target commands, whereas direct-injection baselines learn thermal representations and grounding from limited action supervision (Section 4.3; Appendix A.2 and B). Table 2 establishes the effectiveness of the complete system, but leaves its gains confounded with additional pretrained reasoning capability and an easier control interface. Matched controls are particularly important for the fixed-position, two-candidate tasks.

Questions for authors
  1. Implementation clarification: Does CAG use consecutive-frame displacement or displacement from a no-contact reference? Please reconcile Equation (7), Appendix D.1, and the supplementary video's initial/current marker visualization at 00:28. Can you provide the implemented rule and gate traces during stable holding and release? This would distinguish a reporting issue from a limitation of the implemented contact criterion.

  2. Evidence needed for the current contribution claim: How does performance compare with a simple thermal-region selector connected to the same controller, and with baselines supplied equivalent target information or auxiliary grounding supervision? These controls would materially affect the assessment of whether the advantage comes from the proposed decomposition. This request concerns the present attribution claim, rather than a general suggestion for future task expansion.

Reviewer_3c2mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Thermo-VTLA separates physical-state grounding from contact-rich manipulation. At episode initialization, a frozen VLM processes RGB–thermal images and selects a spatially explicit command from predefined candidates, e.g. translating “pick the cold Coke” into “pick the right Coke.” A SmolVLA-based controller then generates action chunks using this command, RGB observations, robot state, and tactile feedback, without thermal input. A frozen VQ-VAE encodes tactile images; contact-aware activation and learned gated fusion regulate their contribution to control. The main contribution is an interpretable hierarchy that decouples thermal–language grounding from low-level action learning while retaining tactile feedback during execution. Experiments report 93.3% average success across three physical tasks.

Critical strengths

An interesting issue. I think the authors find a problem that make sense in many industrial scenes. So I think this paper is quite informativeness.

The decomposition matches the complementary roles of thermal and tactile sensing and provides an interpretable interface between perception and control.

Real-robot experiments report 93.3% average success, with ablations supporting the benefits of thermal grounding and tactile integration.

Critical weaknesses

The comparisons do not isolate architectural advantages from pretrained knowledge: the proposed method uses a powerful frozen VLM, whereas direct-injection baselines learn thermal alignment from limited demonstrations. A simple thermal detector plus the same controller is missing.

Experiment settings are quite simple. Evaluation covers only three fixed-position, binary-selection tasks with 20 trials each. Generalization across thermal conditions and the necessity of VLM reasoning remain unestablished.

Questions for authors
  1. Can you compare against a simple thermal-threshold target selector coupled to the identical VTLA controller? This would clarify whether pretrained VLM reasoning is necessary for the evaluated binary-selection tasks.

  2. Equation 7 measures consecutive-frame marker motion. How does contact-aware gating preserve tactile feedback during sustained static contact? Please clarify the implemented reference and report contact-detection accuracy.

  3. How do grounding accuracy and task success change under weak thermal contrast, ambient-temperature shifts, and scenes with more than two candidates? These tests would substantiate the claimed thermal generalization.

Reviewer_WAT2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Thermo-VTLA: an off-the-shelf VLM (GPT-5.6-sol) looks at an RGB and a thermal image once per episode and rewrites an instruction, e.g., turning "pick up the cold Coke" into "pick up the coke on the right", which a VLA with added tactile input executes. It is evaluated on three real-robot tasks, each with two visually identical objects that differ only in temperature state.

Critical strengths
  • Using thermal sensing for pre-contact physical state is a relevant practical problem.
  • The tactile ablation, where all variants receive the correct prompt, shows that touch improves execution.
Critical weaknesses
  • The contribution is questionable, as thermal is only used to turn the instruction into a left/right prompt, and a binary choice between two objects with a clear temperature difference doesn't need a frontier VLM. A simple pre-processing of the thermal image (e.g., comparing the temperature of the two candidate regions) would likely give the same result. I understand this is more general, but the current experiments provide little insight. In general, the results are not surprising. The two candidate objects look identical in RGB by design, so without thermal information the baselines in Table 1 can only guess. The baselines that do use thermal in Table 2 are all on par with random choice, and it's not clear whether their thermal encoder actually learned to use the signal at all.
  • The tactile integration in VLAs is not novel (plenty of references in the paper itself), and it's not been compared with existing VTLA work.
Questions for authors
  1. How does a simple thermal heuristic perform for prompt selection compared with GPT-5.6-sol?
  2. How does the tactile integration compare against the VTLA methods cited in the paper?
  3. In the direct thermal-injection baseline, the thermal encoder is initialized from the RGB encoder and trained on 300 demonstrations. Why does this fail? Have you tried alternatives such as extracting simpler features from thermal image regions and feeding them to the model?
  4. All tasks involve two candidates in fixed left/right positions. How would the approach handle more objects, arbitrary positions, or physical attributes that can't be expressed as a spatial prompt?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.