- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes FusionCLIP, a CLIP-style contrastive pretraining framework for paired visible-light videos and multivariate diagnostic signals from tokamak discharges. The video branch uses Spatiotemporal Stability Selection (SSS) to keep a subset of patch tokens. The signal branch uses a Temporal Query Encoder (TQE) that reads irregularly sampled signals at their timestamps. The authors build EAST-MMD (1307 EAST discharges), pretrain on 667 of them, and fine-tune on disruption prediction, ELM classification and H98 regression. They report the best aggregate score (89.07%) against general-purpose pretrained backbones and a CLIP-Base variant, with ablations and attention maps.
I will be happy to raise my scores upon clarification.
Critical strengths
I like the domain and I believe the problem is well chosen. Pairing video and diagnostic signals for contrastive pretraining is an interesting fusion, and handling signals sampled from 25 Hz to 250 kHz without resampling is a real issue. The paper also tests three different downstream tasks and includes ablations, which is more than many applied papers do.
Critical weaknesses
-
The evidence for the architecture is thin. Against CLIP-Base (same data, same pretraining), the gain is mostly on disruption prediction. On ELM and H98 the two are essentially tied or mixed (CLIP-Base has higher ELM ACC and AUC and higher H98 R2). The disruption test set has about 24 disruptive discharges (TPR of 95.83% vs 91.67% is one discharge). With no seeds, no confidence intervals and no significance test, the 2.12 point aggregate gain over CLIP-Base cannot be separated from noise. The large gap to the general-purpose baselines is also confounded, since those use a TCN for signals while the others use TQE, so it mostly shows that in-domain data and TQE help, not that the contrastive objective or SSS does.
-
SSS is heuristic and weakly justified. The Fourier transform is applied along the feature dimension (Eq. 3), but channel order in a transformer embedding has no meaning, so a low-pass filter there has no clear physical interpretation and depends on an arbitrary channel ordering. The fusion weights (0.20/0.35/0.45), the 1:2:1 temporal kernel and the 50% mask ratio are not motivated to the best of my knowledge. Table 3 selects the mask ratio from downstream results, and it is unclear whether this was done on validation or test data. Fig. 3 shows that FusionCLIP masks background, which is expected by construction, so it does not show that the selection is correct.
-
Presentation and consistency. Terms such as "diagnostic systems" and "plasma radiation structures" are introduced without concrete examples (time series? 2D images?), and the related work does not say how FusionCLIP differs from the cited work (e.g. DIII-D line 117). Notation is heavy and the video resolution and tensor dimensions are not given. Table 1 sits out of place and its caption says almost nothing (1 line). The CLIP-Base row in Table 1 (ELM ACC 91.56%) does not match the "Full Token" row in Table 3 (91.18%) although the other numbers are identical.
Questions for authors
- Is the mask ratio and the SSS weighting chosen on validation data only? Can you report mean and std over several seeds, at least for the CLIP-Base vs FusionCLIP comparison?
- How are the two encoder outputs used downstream (CLS of each, concatenated, averaged)?
- Do all 38 variables share one event-type alphabet, and is a variable absent in some discharges?
- What is the video resolution and the tensor shape entering the encoder?
- Why did you choose an early summation vs not a concatenation of the embeddings in Eq. 7. ? See (Math et al., 2026, Context-Informed Sequence Classification: A Multimodal Approach to Vehicle Diagnostics, ICLR Workshop on Time Series in the Age of Large Models)
- Why didn't you consider cross-modality fusion using the two encoders to learn early the correlation between the diagnostic data and the plasma discharges ? e.g., cross-attention, co-attention etc.. Even if modalities are unaligned, this should be possible.
- Did you do some ablations regarding the time information added at the input level and in the attention score ?