acceptodds
Withdrawn from ICLR 2027

FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visible-light videos and multivariate diagnostic signals provide complementary observations of tokamak discharges. However, existing fusion pretraining methods, primarily learn representations by reconstructing or predicting observations, leaving the correspondence between these modalities underexplored. Representation learning is further challenged by the fact that plasma-related regions occupy only part of each video frame and diagnostic variables have substantially different sampling rates. We propose FusionCLIP, a multimodal contrastive pretraining framework. Spatiotemporal Stability Selection retains informative visual tokens, while a Temporal Query Encoder integrates asynchronous diagnostic signals using their observation timestamps. Contrastive learning then aligns the two modality representations. We construct EAST Multimodal Diagnostics (EAST-MMD), comprising 1307 EAST discharges, and evaluate the model on three independently fine-tuned tasks: disruption prediction, edge-localized mode (ELM) classification, and regression. FusionCLIP achieves an aggregate score of 89.07%, exceeding the best general-purpose pretraining configuration and an in-domain contrastive pretraining configuration without Spatiotemporal Stability Selection by 6.27 and 2.12 percentage points, respectively. Ablations and attention visualizations further support the contributions of its key components.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Nuclear Fusion, Contrastive Learning, Multimodal Learning, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aq4kOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FusionCLIP, a CLIP-style contrastive pretraining framework for paired visible-light videos and multivariate diagnostic signals from tokamak discharges. The video branch uses Spatiotemporal Stability Selection (SSS) to keep a subset of patch tokens. The signal branch uses a Temporal Query Encoder (TQE) that reads irregularly sampled signals at their timestamps. The authors build EAST-MMD (1307 EAST discharges), pretrain on 667 of them, and fine-tune on disruption prediction, ELM classification and H98 regression. They report the best aggregate score (89.07%) against general-purpose pretrained backbones and a CLIP-Base variant, with ablations and attention maps.

I will be happy to raise my scores upon clarification.

Critical strengths

I like the domain and I believe the problem is well chosen. Pairing video and diagnostic signals for contrastive pretraining is an interesting fusion, and handling signals sampled from 25 Hz to 250 kHz without resampling is a real issue. The paper also tests three different downstream tasks and includes ablations, which is more than many applied papers do.

Critical weaknesses
  • The evidence for the architecture is thin. Against CLIP-Base (same data, same pretraining), the gain is mostly on disruption prediction. On ELM and H98 the two are essentially tied or mixed (CLIP-Base has higher ELM ACC and AUC and higher H98 R2). The disruption test set has about 24 disruptive discharges (TPR of 95.83% vs 91.67% is one discharge). With no seeds, no confidence intervals and no significance test, the 2.12 point aggregate gain over CLIP-Base cannot be separated from noise. The large gap to the general-purpose baselines is also confounded, since those use a TCN for signals while the others use TQE, so it mostly shows that in-domain data and TQE help, not that the contrastive objective or SSS does.

  • SSS is heuristic and weakly justified. The Fourier transform is applied along the feature dimension (Eq. 3), but channel order in a transformer embedding has no meaning, so a low-pass filter there has no clear physical interpretation and depends on an arbitrary channel ordering. The fusion weights (0.20/0.35/0.45), the 1:2:1 temporal kernel and the 50% mask ratio are not motivated to the best of my knowledge. Table 3 selects the mask ratio from downstream results, and it is unclear whether this was done on validation or test data. Fig. 3 shows that FusionCLIP masks background, which is expected by construction, so it does not show that the selection is correct.

  • Presentation and consistency. Terms such as "diagnostic systems" and "plasma radiation structures" are introduced without concrete examples (time series? 2D images?), and the related work does not say how FusionCLIP differs from the cited work (e.g. DIII-D line 117). Notation is heavy (Oi,m,j)(O_{i,m,j}) and the video resolution and tensor dimensions are not given. Table 1 sits out of place and its caption says almost nothing (1 line). The CLIP-Base row in Table 1 (ELM ACC 91.56%) does not match the "Full Token" row in Table 3 (91.18%) although the other numbers are identical.

Questions for authors
  • Is the mask ratio and the SSS weighting chosen on validation data only? Can you report mean and std over several seeds, at least for the CLIP-Base vs FusionCLIP comparison?
  • How are the two encoder outputs used downstream (CLS of each, concatenated, averaged)?
  • Do all 38 variables share one event-type alphabet, and is a variable absent in some discharges?
  • What is the video resolution and the tensor shape entering the encoder?
  • Why did you choose an early summation vs not a concatenation of the embeddings in Eq. 7. ? See (Math et al., 2026, Context-Informed Sequence Classification: A Multimodal Approach to Vehicle Diagnostics, ICLR Workshop on Time Series in the Age of Large Models)
  • Why didn't you consider cross-modality fusion using the two encoders to learn early the correlation between the diagnostic data and the plasma discharges ? e.g., cross-attention, co-attention etc.. Even if modalities are unaligned, this should be possible.
  • Did you do some ablations regarding the time information added at the input level and in the attention score ?
Reviewer_pf6wOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposed FusionCLIP for multimodal representation learning in fusion plasma diagnostics. The main idea is to align the visible-light videos and diagnostic time-series signals with contrastive pretraining. The authors introduced a spatiotemporal token selection method to preserve informative visual regions and a temporal query encoder to handle diagnostic signals with different sampling rates. Experiments demonstrated the effectiveness of the proposed method on downstream tasks.

Critical strengths
  1. Multimodal representation learning for fusion plasma diagnostics is an interesting problem. The videos and diagnostic signals contain complementary information, and it is meaningful to learn a shared representation for different downstream tasks.
  2. The temporal query encoder is reasonable. Since the diagnostic signals have very different sampling frequencies, directly resampling them may lose important temporal information. The proposed method provides an interesting solution to this problem.
  3. The experimental results are promising. Compared with baseline methods, FusionCLIP achieved competitive results on disruption prediction, ELM classification, and H98 regression.
  4. The construction of the EAST-MMD dataset is also valuable. The dataset can support further research on multimodal learning for fusion diagnostics.
Critical weaknesses
  1. My main concern is the novelty. The proposed framework mainly combines contrastive learning, visual token selection, and temporal attention. These techniques have been widely investigated in multimodal representation learning. It is unclear what the main methodological contribution is beyond their application to fusion plasma diagnostics.
  2. The motivation of the spatiotemporal stability selection is not convincing enough. This work used fourier filtering along the feature dimensions to identify stable visual tokens. However, the feature dimensions do not necessarily correspond to meaningful frequency components. Therefore, I am not convinced that this strategy can really identify informative plasma regions.
  3. The experiments are only conducted on the EAST dataset. Although the model is evaluated on different downstream tasks, this does not necessarily demonstrate that the learned representation can generalize to other tokamaks or diagnostic systems.
Questions for authors
  1. Please clarify the main methodological novelty compared with existing multimodal contrastive learning and irregular time-series modelling methods. 2.Explain why Fourier filtering along feature dimensions can represent visual feature stability, and discuss the assumptions and limitations of this design. 3, Please also describe the effect of highly overlapping training windows and avoid over-claimed statements about the generalization of the learned representations.
Reviewer_QjAbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FusionCLIP, a multimodal contrastive pretraining framework for tokamak plasma diagnostics. The main idea is to learn transferable representations by aligning visible-light plasma videos with multivariate diagnostic time series. The proposed framework includes Spatiotemporal Stability Selection (SSS), which selects informative visual tokens, and a Temporal Query Encoder (TQE), which handles diagnostic signals with different sampling rates using their timestamps. The authors construct the EAST-MMD dataset with 1,307 discharges and evaluate the pretrained representations on three downstream tasks: disruption prediction, ELM classification, and H98 regression. FusionCLIP achieves a Joint score of 89.07%, compared to 86.95% for the in-domain CLIP-Base baseline. The paper also includes component ablations and attention visualizations to support the proposed design.

Critical strengths
  1. An interesting multimodal pretraining setup for fusion diagnostics.

I like the motivation behind aligning plasma videos with diagnostic signals instead of learning representations from each modality separately. The two modalities clearly provide complementary information, and the different sampling rates make this a non-trivial problem. The construction of EAST-MMD is also a useful contribution. I particularly appreciate that the authors use discharge-level splits, which is important given the substantial overlap between neighboring windows.

  1. Promising improvements supported by useful ablations.

The results suggest that the proposed pretraining strategy can improve downstream performance, particularly for disruption prediction. The comparison with CLIP-Base is useful because it helps isolate the contribution of SSS. The token-selection experiments in Table 3 are also encouraging: SSS at a 50% mask ratio performs better than both full-token processing and random token selection at the same ratio. This provides some evidence that the selection strategy is doing more than simply reducing the number of tokens.

Critical weaknesses
  1. The experimental improvements need stronger evidence of robustness.

While the overall results are promising, the improvement over the closest in-domain baseline is relatively modest. The Joint score increases from 86.95% to 89.07%, and much of this gain comes from disruption prediction. For ELM classification, the macro-F1 improvement is only 0.64 percentage points, while CLIP-Base actually achieves a slightly better H98 R2.

More importantly, the disruption test set contains only 97 discharges, and the reported TPR/FPR values suggest that only 24 of them are disruptive. Compared with CLIP-Base, the improvement in disruption F1 appears to correspond to identifying one additional disruptive discharge and avoiding two false alarms. These are useful improvements, but with such a small evaluation set, I would like to see uncertainty estimates or results from multiple runs before drawing a strong conclusion.

The evaluation is also limited to a single tokamak. This is understandable for an initial study, but it leaves open how transferable these representations really are beyond the current dataset.

  1. Some methodological choices are insufficiently justified, and the comparisons do not fully isolate their contributions.

My main concern is with the motivation and implementation of the feature-stability component in SSS. In Eq. (4), the denominator appears to use the difference between filtered and original features without taking its absolute value. If this is the intended implementation, it is unclear how negative differences are handled or whether the resulting score really measures stability as described.

I am also not fully convinced by applying Fourier low-pass filtering along the learned feature dimension. Unlike a spatial or temporal axis, embedding channels do not have an obvious frequency ordering. The attention visualizations are suggestive, but they do not establish that this particular stability measure reliably identifies informative plasma regions.

For TQE, the paper would benefit from a more controlled comparison with alternative methods for irregularly sampled time series. Table 2 is helpful, but comparing TQE against a conventional temporal backbone without pretraining does not fully establish whether timestamp-aware query encoding is preferable to other time-aware alternatives under the same pretraining conditions.

Questions for authors
  1. Clarification regarding disruption prediction and warning times.

Each input window covers 50 ms, but a successful warning only needs to precede disruption by 10 ms. Is the reported alarm timestamp associated with the beginning or the end of the input window? How do you ensure that a warning does not use observations occurring after its reported timestamp? This matters for interpreting the early-warning results and the example in Figure 4.

  1. Clarification regarding the feature-stability formulation.

In Eq. (4), should the denominator contain the absolute difference between the original and filtered features? If the equation is correct as written, could you explain how the sign of the difference affects the voting mechanism? I would also be interested to know whether the results are sensitive to permutations of the embedding channels, given that the Fourier transform is applied along the feature dimension.

  1. Robustness of the reported improvements.

Could you report results across multiple random seeds, or discharge-level confidence intervals for the main baselines? In particular, I would like to know whether the disruption prediction improvement remains consistent given the relatively small number of disruptive discharges in the test set. This would materially strengthen the empirical conclusions.

  1. Comparison with alternative temporal encoders.

Could the authors provide a matched comparison between TQE and an established timestamp-aware encoder, or a resampling-based encoder trained with the same contrastive objective and visual backbone? This would help clarify how much of the gain comes specifically from TQE rather than the overall pretraining setup.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.