acceptodds
Under review as a conference paper at ICLR 2027

READit: Rapid and Efficient Attention-based Distillation for Irregularly Sampled Light Curves

Abstract

Next-generation astronomical surveys like the Vera C. Rubin Observatory (LSST) will accumulate petabyte-scale data over the next decade, producing tens of terabytes each night, yet extracting representations via self-supervised learning currently demands prohibitive pre-training compute. To address this bottleneck, we introduce *READit*, an attention-based foundation model optimized via student–teacher self-distillation for asynchronous, multi-band light curves. Pre-trained on million Zwicky Transient Facility (ZTF) light curves, *READit* reaches peak representational convergence in just 11 epochs ( GPU-hours)—a compute reduction ( speedup) over prior multi-band contrastive frameworks. This rapid optimization is achieved by coupling multi-crop self-distillation with physics-informed temporal augmentations that stabilize the learning trajectory. On downstream benchmarks, *READit* consistently outperforms existing astronomical foundation models, general time-series architectures, and classical statistical baselines on 12-class variability classification. Moreover, *READit* exhibits remarkable label efficiency in extreme low-data regimes, clusters unseen out-of-distribution classes, and demonstrates robust zero-shot and few-shot cross-survey transfer to simulated LSST observations (PLAsTiCC). Consequently, *READit* establishes a computationally sustainable, "Green AI" foundation model for survey-scale time-domain discovery.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-supervised learning, Representation learning, Time series analysis, Light curves, Foundation models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_f8NBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces READit, a self-supervised representation learning model for irregularly sampled, multiband astronomical light curves. It applies DINO-style self-distillation to a compact Transformer using multi-scale temporal views and observation-specific data augmentations. Experiments on ZTF light curves report improved classification performance and faster pre-training convergence over existing methods, while also evaluating label efficiency and transfer to unseen classes and simulated LSST observations.

Critical strengths
  1. The paper applies DINO-style self-distillation to irregular multiband light curves, improving classification performance over the contrastive learning baseline Astra-CLR. It also achieves better performance than supervised training from scratch in low-label regimes, demonstrating the potential of self-distillation for representation learning in this domain.

  2. The evaluation extends beyond in-distribution classification to unseen astronomical classes and simulated LSST observations (PLAsTiCC). The few-shot results suggest that representations learned from ZTF observations can transfer to new astronomical classes and observational settings.

  3. The paper provides ablations of observation-specific augmentations and multi-crop configurations. The results demonstrate the benefits of temporal masking and photometric perturbations, as well as combining global and local views rather than using only global views (Table 1).

Critical weaknesses
  1. The comparison with existing contrastive learning approaches is insufficient. READit combines existing light-curve preprocessing and multi-crop strategies with DINO-style self-distillation. Although several design choices are examined through ablations, no controlled comparison of learning methods using the same data and backbone is provided, making it difficult to isolate the contribution of self-distillation. Additionally, the cross-survey transfer experiments compare only against random initialization, leaving the advantage over existing pre-trained methods unclear.

  2. The claimed computational savings are not supported by a direct comparison of computational costs. The reported 18× speedup and approximately 95% compute reduction are based on comparing 200 epochs for Astra-CLR with 11 epochs for READit (Figure 4a). However, the methods differ in their training configurations and computational cost per epoch, making it difficult to establish the claimed reduction without a direct comparison in GPU-hours or FLOPs.

  3. The construction of the evaluation dataset is unclear. Appendix A.1 states that 120K-Eval contains 10,000 light curves for each of 12 classes. However, according to Figure 5, some rare classes contain fewer than 10,000 samples even in the full dataset. For example, CEP accounts for only 0.059% of 2.6M-Full, corresponding to approximately 1,530 sources. It is therefore unclear how the balanced 120K-Eval dataset was constructed.

Questions for authors
  1. How do contrastive learning and DINO-style self-distillation compare in classification performance and convergence speed when using the same data and backbone, with configurations appropriate for each learning method?

  2. What is the actual computational speedup over Astra-CLR when measured in GPU-hours or FLOPs rather than pre-training epochs?

  3. How were 10,000 samples per class obtained for the rare classes in 120K-Eval? If oversampling was used, were training and evaluation sets separated at the unique-source level, and how does performance compare when evaluated on unique sources without duplication?

5 comments
AuthorsOpenReview ↗

Response to Reviewer f8NB

We thank Reviewer f8NB for their rigorous evaluation of our methodology, computational metrics, and dataset construction.

Regarding Pre-Trained Baselines on PLAsTiCC: We have updated Table 4 (Section 5.2) to directly benchmark against pre-trained Astra-CLR weights on PLAsTiCC. READit strictly outperforms pre-trained Astra-CLR across both Domain Adaptation (Tier 1: 60.16%60.16\% vs. 53.59%53.59\% 5-shot F1) and Comprehensive Transfer (Tier 2: 48.73%48.73\% vs. 47.14%47.14\% 5-shot F1). Please see Point 3 in our General Response.

Q1 & Q2: Controlled Comparison & Actual Hardware Speedup: To isolate the learning objective on identical hardware (2×2\times NVIDIA T4 GPUs), we conducted a controlled ablation on the 26K-Proxy dataset using the exact same Tiny backbone (L=4,d=256L=4, d=256) and 2 global views (Table 8, Appendix F). Self-distillation reaches its loss basin in 0.380.38 GPU-hours, whereas the contrastive baseline (NT-Xent) requires 7.657.65 GPU-hours—an empirical 19.9×19.9\times wall-clock speedup (95.0%95.0\% compute reduction). Furthermore, contrastive learning requires 4–5×4\text{--}5\times more compute simply to match READit's early performance. Please see Point 1 in our General Response for full details.

Q3: Dataset Construction & Zero-Leakage Clarification: We guarantee zero data leakage at the unique-source level. Unique physical sources were strictly partitioned into disjoint training (80%) and validation (20%) splits prior to independent minority-class oversampling. Validation metrics are evaluated strictly on physical sources isolated from training. We have clarified this protocol in Appendix A.1 (see Point 4 in our General Response).

AuthorsOpenReview ↗

Response to Reviewer f8NB

Thank you again for your thorough review. To ensure our revisions are directly accessible within this thread, we have included our General Response below and updated our revised PDF (with all modified passages highlighted in blue).

In addition to addressing your specific questions regarding the controlled objective comparison (Appendix F, Table 8), pre-trained PLAsTiCC baselines (Section 5.2, Table 4), and strict unique-source segregation in 120K-Eval (Appendix A.1), we have also added a comprehensive benchmark on the human-vetted StarEmbed (ZTF-40K) catalog (Appendix E) along with direct comparisons against all baseline models on this catalog (Appendix G, Table 9), where READit demonstrates strong, zero-leakage performance.


General Response: Summary of Revisions & New Controlled Experiments

We thank the reviewers for their constructive feedback. We have revised our manuscript (modifications highlighted in blue in the updated PDF). Below is an executive summary of our new empirical additions and clarifications:

1. Technical Novelty & Controlled Compute Speedup (Appendix F, Table 8): To isolate the self-distillation objective from architectural choices, we executed a strictly controlled ablation on identical hardware (2×2\times NVIDIA T4 GPUs) on 26K-Proxy, fixing the Tiny Transformer backbone (L=4,d=256L=4, d=256), 2 global views, and Cosine scheduler:

  • 19.9×19.9\times Wall-Clock Speedup: Self-distillation reaches its loss basin in 0.380.38 GPU-hours (∼\simEpoch 4) vs. 7.657.65 GPU-hours (Epoch 123) for contrastive NT-Xent—an empirical 95.0%95.0\% compute reduction.
  • Time-to-Parity: Contrastive learning requires 4–5×4\text{--}5\times more compute (∼1.5\sim 1.5h) merely to match the 0.40650.4065 Macro-F1 that READit achieves in 2323 minutes, requiring +6.15+6.15 additional GPU-hours for a marginal +0.028+0.028 gain.
  • Overcoming 2-View Ceilings via Temporal Anchoring: While 2-view distillation converges rapidly, it imposes an information bottleneck. READit introduces Temporal Anchoring (3 deterministic global views spanning start, mid, end + 6 multi-scale local views), raising downstream performance to ∼0.425\sim 0.425 Macro-F1 in just 2.622.62 GPU-hours (∼\simEpoch 8, Figure 11).
  • O(N)\mathcal{O}(N) Linear Memory vs. O(N2)\mathcal{O}(N^2) OOM: Crucially, scaling to 9 views is computationally feasible only under self-distillation due to linear memory scaling O(N)\mathcal{O}(N). An equivalent 9-view contrastive setup triggers immediate Out-of-Memory (OOM) failures due to quadratic O(N2)\mathcal{O}(N^2) negative-pair matrices.

2. Expert-Vetted StarEmbed (ZTF-40K) Benchmark (Appendix E): We evaluated READit on the human-vetted StarEmbed benchmark. Using a 3D spherical kk-d tree matching radius (Δθ≥1.5′′\Delta\theta \ge 1.5''), we isolated a strictly non-overlapping, zero-leakage test set (N=3,487N=3{,}487). READit achieves virtually identical Linear Probing Macro-F1 on this clean split (0.6418±0.00600.6418 \pm 0.0060) compared to the full test set (0.6581±0.00460.6581 \pm 0.0046), ruling out pre-training memorization, and demonstrates competitive performance against all baseline models on this catalog (Appendix G, Table 9).

3. Cross-Survey Transfer vs. Pre-Trained Baselines (Section 5.2, Table 4): On simulated LSST data (PLAsTiCC), READit strictly outperforms pre-trained Astra-CLR weights across all regimes:

  • Tier 1 (Domain Shift, 3-Way): 75.31%75.31\% vs. 64.06%64.06\% Recall@1; 60.16%60.16\% vs. 53.59%53.59\% 5-shot Macro-F1.
  • Tier 2 (Comprehensive Transfer with Supernovae, 5-Way): 71.72%71.72\% vs. 61.09%61.09\% Recall@1; 48.73%48.73\% vs. 47.14%47.14\% 5-shot Macro-F1.

4. Dataset Construction & Zero-Leakage Clarification (Appendix A.1): In 120K-Eval, unique physical sources were strictly partitioned into disjoint training (80%) and validation (20%) splits prior to independent oversampling, guaranteeing zero source-level leakage.

5. Downstream Classifier Configurations (Appendix C): We detailed classifier hyperparameters (Linear Probe: lbfgs, C=1.0C=1.0, max_iter=4000; kk-NN: k=20,τ=0.07k=20, \tau=0.07; RF: 200 trees, depth 20) and ASTROMER2/CHRONOS2 embedding extraction protocols.


Thank you for the detailed responses and additional experiments. The new results and clarifications have largely addressed my concerns regarding Q1, Q2, and Q3.

The controlled comparison (Table 8) shows that DINO-style self-distillation converges much faster, and the ablations (Table 1) show that multi-crop views improve representation quality. However, both are established techniques.

The additional gains from Temporal Anchoring and the specific 9-view configuration appear relatively modest: Temporal Anchoring vs. stochastic global views (0.383 vs. 0.378) and 9 vs. 6 views (0.383 vs. 0.381). While the manuscript motivates these design choices, the empirical analysis does not clearly establish how much they contribute to the improvements over prior methods.

I therefore remain unconvinced of the methodological contribution and will maintain my original rating.

Reviewer_d5bcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a foundation model for astronomical data. The model READit is trained with DINO and customized application specific physical augmentation, and it shows enhanced label efficient, fast convergence, and generalization.

Critical strengths
  1. The paper proposes the foundation models for astronomical data, which is relevant to the astronomy community.
  2. The paper shows improved computational efficiency.
Critical weaknesses
  1. The main concern is the technical novelty of the work. The work appears to be using DINO SSL architecture with domain-specific augmentations and preprocessing. The generalizability, label efficiency, and computational efficiency have been well established in prior literature.
  2. The computational efficiency claim require further justification. The paper compares epochs to convergence, which does not establish equivalent reductions in computations as different objectives and architectures could differ. Moreover, comparing with the medium-sized models could help as medium-sized models could have greater computational efficiency with similar performance or minor degradation. Matched GPU-hours or flops could enhance the claim.
Questions for authors
  1. Could a smaller backbone achieve comparable performance with lower computational cost, given the early saturation in downstream performance?
  2. Masked reconstruction has been an alternative to contrastive learning, have the authors considered MAE approaches that do not require additional branch for extra compute?
  3. Can authors elaborate on the technical novelty of the work?
2 comments
AuthorsOpenReview ↗

Response to Reviewer d5bc

We thank Reviewer d5bc for their constructive suggestions regarding model scaling and computational validation.

Regarding Medium-Sized Model Comparison: In our original submission, we evaluated architectural scaling in Ablation Suite D (Table 1, Figure 9) on the 6K-Proxy dataset, directly comparing our Tiny backbone (L=4,d=256L=4, d=256) against a Medium backbone (L=6,d=384L=6, d=384). The Tiny backbone trained stably across all 35 epochs (mean epoch duration of ∼274\sim 274s), where downstream Macro-F1 steadily improved to 0.3830.383 in lockstep with decreasing validation loss. In contrast, the Medium backbone increased epoch duration by 2.5×2.5\times (∼704\sim 704s) and suffered from premature optimization divergence: its validation loss hit a minimum early at Epoch 10, after which validation loss rose while downstream metrics decoupled (exhibiting shortcut-like memorization), yielding an early-stopping Macro-F1 of only 0.3780.378. This empirical comparison confirms that scaling capacity beyond our compact architecture provides no representational advantage, but instead accelerates overfitting and representational instability on the astronomical manifold while significantly increasing compute cost.

Q1: Could a smaller backbone achieve comparable performance? Our proposed 4-layer backbone is already compact (d=256,L=4d=256, L=4). Downscaling further (e.g., to 2 layers) restricts the receptive field required to attend across 600-step multi-band sequences. As shown in our scaling ablation (Suite D), our Tiny configuration represents the optimal capacity floor.

Q2: Masked Autoencoding (MAE) vs. Self-Distillation: While MAE avoids a momentum branch, pixel/flux-level reconstruction in astronomical time-series forces the encoder to fit heteroscedastic photometric noise and observational cadence gaps. In contrast, self-distillation optimizes high-level semantic distributions in latent space, providing natural invariance to noise. Furthermore, the Teacher branch in DINO requires zero backpropagation or gradient computation (it is updated via EMA with negligible memory cost). As shown in our controlled experiments, the primary computational overhead in self-supervised learning is negative-pair calculation, which self-distillation completely eliminates.

Q3: Technical Novelty: Please refer to Point 1 in our General Response, where we outline our core technical contributions: co-designing an O(N)\mathcal{O}(N) linear-memory distillation objective with Temporal Anchoring to break the 2-view capacity ceiling without Out-Of-Memory (OOM) failures.

Thank you for the responses and clarification. The comparison provides useful insights into scaling, but it does not establish whether a smaller backbone could achieve comparable performance at lower cost or justify the claim of an optimal capacity floor. The additional wall-clock and time-to-parity comparisons help address my computational-efficiency concern, but the absence of teacher backpropagation does not eliminate the cost of teacher forward passes. My main concern about technical novelty remains: most of the design choices appear to be incremental or already established. While I recognize the application-specific benefits, I am still not convinced that it establishes sufficient methodological novelty.

Reviewer_Jch1OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes READit, a self-supervised method for learning representations of irregularly sampled, multi-band astronomical light curves. It combines DINO-style self-distillation with domain-specific data augmentations. Experiments show improved classification performance over existing baselines, along with promising label efficiency and cross-survey transfer. The authors also report substantially faster pretraining compared with Astra-CLR.

Critical strengths
  1. The experimental scope is broad, and the empirical results are promising. The paper evaluates the method across downstream classification, label efficiency, OOD generalization, and cross-survey transfer. The paper also achieves strong performance compared with the reported baselines.
Critical weaknesses
  1. Distinction between the proposed method and prior work could be clearer. My reading is that the methodology is largely Astra-CLR (e.g., the preprocessing and view construction in sec 3.1) combined with DINO-style self-distillation. For example, after introducing prior work, line 93–98 describe the proposed method without clearly stating what bottleneck in previous methods it addresses or which technical components are new other than training efficiency (which I also have some concerns, see the second point below.)

I have no doubt that making this combination work requires substantial engineering effort, and the empirical results are promising. However, I would appreciate a clearer explanation of the contribution and novelty beyond combining Astra-CLR with DINO.

  1. I am not sure that the computational-efficiency comparison in fig. 4(a) is fair enough to support the claimed 18× speedup. Although READit and Astra-CLR use the same underlying ZTF data distribution, several other factors differ, including the view-generation procedure, model hidden dimension, and optimization schedule. It is therefore unclear how much of the faster convergence comes from the self-distillation objective itself.

A controlled comparison would help. For example, the authors could keep the backbone, views, and optimization settings fixed and compare DINO-style self-distillation with the contrastive objective used by Astra-CLR. Some hyperparameter tuning for both objectives would also help rule out undertraining of either method.

More importantly, the Astra-CLR paper states that the model was trained for exactly 200 epochs, but I could not find evidence that it "required 200 epochs to converge" as the author claim. READit instead identifies its best downstream checkpoint at Epoch 11. Could the authors clarify the basis for the statement that "Astra-CLR requires 200 epochs to converge"? Reporting downstream performance versus training timeboth methods would make the efficiency claim more convincing.

Questions for authors
  1. I suspect the authors chose Gaia DR3 machine-generated labels because of their larger scale. Could the authors clarify this choice? Since these labels may contain some classification errors, it might also be interesting to evaluate on expert-vetted catalog such as [1], while checking for potential source overlap with READit's pretraining data.

  2. In Line 96, what prior work does “regularized interpolation” refer to?

  3. Did the authors tune hyperparameters for the downstream classifiers, including the linear probe [2, 3], kNN, and random forest [4]? A small HPO (e.g. learning rate, max_depth for RF...) would make the comparison more convincing. (ICML) 2026.


[1] Li W, Chen HY, Rehemtulla N, Shah VG, Kim D, Wu D, Lin Q, Miller AA, Liu H. StarEmbed: benchmarking time series foundation models on astronomical observations of variable stars.

[2] Caron M, Touvron H, Misra I, Jégou H, Mairal J, Bojanowski P, Joulin A. Emerging properties in self-supervised vision transformers. In2021 IEEE/CVF international conference on computer vision (ICCV) 2021.

[3] Oquab M, Darcet T, Moutakanni T, Vo H, Szafraniec M, Khalidov V, Fernandez P, Haziza D, Massa F, El-Nouby A, Assran M. Dinov2: Learning robust visual features without supervision.

[4] Hosenie Z, Lyon RJ, Stappers BW, Mootoovaloo A. Comparing multiclass, binary, and hierarchical machine learning classification schemes for variable stars. Monthly Notices of the Royal Astronomical Society. 2019.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Jch1

We thank Reviewer Jch1 for their constructive feedback and recognition of our broad experimental scope and strong empirical performance.

Q1: Gaia DR3 Labels vs. Expert-Vetted Catalogs & Source Overlap: We chose Gaia DR3 to ensure survey-scale coverage. To address concerns regarding potential label noise and data contamination, we evaluated READit on the human-vetted StarEmbed (ZTF-40K) catalog. We constructed a 3D spherical k-d tree and isolated a strictly non-overlapping, zero-leakage test set (Δθ≥1.5′′\Delta\theta \ge 1.5'', N=3,487N=3{,}487). READit achieves virtually identical Macro-F1 performance on this clean split (0.6418±0.00600.6418 \pm 0.0060) compared to the full test set (0.6581±0.00460.6581 \pm 0.0046), ruling out pre-training memorization. Please refer to Point 2 of our General Response and Appendix E for full details and per-class confusion matrices.

Q2: Clarification on "Regularized Interpolation": We used this term to refer to Gaussian Process (GP) interpolation as utilized in t2t^2 (Allam & McEwen, 2024) to align asynchronous multi-band series onto a uniform grid. We have revised the text in Section 2 to explicitly cite this framework.

Q3: Downstream Classifier Hyperparameters: Yes, downstream classifiers were configured using validated hyperparameters: Linear Probing used the lbfgs solver (C=1.0C=1.0, max_iter=4000, class_weight='balanced'); weighted kk-NN used k=20k=20 with temperature τ=0.07\tau=0.07; and Random Forest was configured with 200 estimators and max depth 20. We have documented these configurations in Appendix C (see Point 5 in our General Response).

Thanks the reviewer for the replies. My concerns regarding Q2 has been resolved.

For Q1, what I would like to see is a comparison with the baselines on the expert-vetted catalogs (Table 2). This would help determine whether the proposed method still outperforms the baselines on these catalogs.

(Minor) For Q3, thanks for clarifying. My remaining concern is whether the conclusions are sensitive to the choices of parameters. A small sensitivity analysis over C for linear probing and k for kNN would strengthen the results, especially since RF hyperparameters were selected via cross-validation.


I also have not seen responses to Weaknesses 1 and 2 raised in my original review, so those concerns remain unresolved.

I will maintain my previous evaluation.

AuthorsOpenReview ↗

Response to Reviewer Jch1

Thank you for reading our reply and engaging in the discussion. We apologize for not copying our responses to Weaknesses 1 and 2 directly into your thread—we detailed both extensively in our General Response and updated PDF (highlighted in blue).


Follow-up on Q3 (Parameter Sensitivity over CC and kk)

We performed a sensitivity analysis on StarEmbed (Test [Filtered]) across C∈{0.1,1.0,10.0}C \in \{0.1, 1.0, 10.0\} and k∈{5,10,20,50}k \in \{5, 10, 20, 50\}:

  • Linear Probe (CC): Macro-F1 remains stable across scales: C=0.1C=0.1 (0.6435 ± 0.0074), C=1.0C=1.0 (0.6418 ± 0.0060), C=10.0C=10.0 (0.6294 ± 0.0030).
  • Weighted kk-NN (kk): Macro-F1 remains consistent: k=5k=5 (0.6331 ± 0.0067), k=10k=10 (0.6415 ± 0.0104), k=20k=20 (0.6262 ± 0.0077), k=50k=50 (0.6079 ± 0.0087). Overall, performance remains robust across orders of magnitude, peaking marginally at C=0.1C=0.1 (0.64350.6435) and k=10k=10 (0.64150.6415).

Follow-up on Q1 (Baselines on StarEmbed)

We have provided a comparison of the baseline models against READit in Appendix G (Table 9), which shows competitive performance for READit. (Note: In the latest PDF, Astra-CLR metrics in Table 9 have been updated to 0.5775 LP / 0.5656 k-NN following resolution of an inference checkpoint path).


General Response: Summary of Revisions & New Controlled Experiments

We thank the reviewers for their constructive feedback. We have revised our manuscript (modifications highlighted in blue in the updated PDF). Below is an executive summary of our new empirical additions and clarifications:

1. Technical Novelty & Controlled Compute Speedup (Appendix F, Table 8): To isolate the self-distillation objective from architectural choices, we executed a strictly controlled ablation on identical hardware (2×2\times NVIDIA T4 GPUs) on 26K-Proxy, fixing the Tiny Transformer backbone (L=4,d=256L=4, d=256), 2 global views, and Cosine scheduler:

  • 19.9×19.9\times Wall-Clock Speedup: Self-distillation reaches its loss basin in 0.380.38 GPU-hours (∼\simEpoch 4) vs. 7.657.65 GPU-hours (Epoch 123) for contrastive NT-Xent—an empirical 95.0%95.0\% compute reduction.
  • Time-to-Parity: Contrastive learning requires 4–5×4\text{--}5\times more compute (∼1.5\sim 1.5h) merely to match the 0.40650.4065 Macro-F1 that READit achieves in 2323 minutes, requiring +6.15+6.15 additional GPU-hours for a marginal +0.028+0.028 gain.
  • Overcoming 2-View Ceilings via Temporal Anchoring: While 2-view distillation converges rapidly, it imposes an information bottleneck. READit introduces Temporal Anchoring (3 deterministic global views spanning start, mid, end + 6 multi-scale local views), raising downstream performance to ∼0.425\sim 0.425 Macro-F1 in just 2.622.62 GPU-hours (∼\simEpoch 8, Figure 11).
  • O(N)\mathcal{O}(N) Linear Memory vs. O(N2)\mathcal{O}(N^2) OOM: Crucially, scaling to 9 views is computationally feasible only under self-distillation due to linear memory scaling O(N)\mathcal{O}(N). An equivalent 9-view contrastive setup triggers immediate Out-of-Memory (OOM) failures due to quadratic O(N2)\mathcal{O}(N^2) negative-pair matrices.

2. Expert-Vetted StarEmbed (ZTF-40K) Benchmark (Appendix E): We evaluated READit on the human-vetted StarEmbed benchmark. Using a 3D spherical kk-d tree matching radius (Δθ≥1.5′′\Delta\theta \ge 1.5''), we isolated a strictly non-overlapping, zero-leakage test set (N=3,487N=3{,}487). READit achieves virtually identical Linear Probing Macro-F1 on this clean split (0.6418±0.00600.6418 \pm 0.0060) compared to the full test set (0.6581±0.00460.6581 \pm 0.0046), ruling out pre-training memorization.

3. Cross-Survey Transfer vs. Pre-Trained Baselines (Section 5.2, Table 4): On simulated LSST data (PLAsTiCC), READit strictly outperforms pre-trained Astra-CLR weights across all regimes:

  • Tier 1 (Domain Shift, 3-Way): 75.31%75.31\% vs. 64.06%64.06\% Recall@1; 60.16%60.16\% vs. 53.59%53.59\% 5-shot Macro-F1.
  • Tier 2 (Comprehensive Transfer with Supernovae, 5-Way): 71.72%71.72\% vs. 61.09%61.09\% Recall@1; 48.73%48.73\% vs. 47.14%47.14\% 5-shot Macro-F1.

4. Dataset Construction & Zero-Leakage Clarification (Appendix A.1): In 120K-Eval, unique physical sources were strictly partitioned into disjoint training (80%) and validation (20%) splits prior to independent oversampling, guaranteeing zero source-level leakage.

5. Downstream Classifier Configurations (Appendix C): We detailed classifier hyperparameters (Linear Probe: lbfgs, C=1.0C=1.0, max_iter=4000; kk-NN: k=20,τ=0.07k=20, \tau=0.07; RF: 200 trees, depth 20) and ASTROMER2/CHRONOS2 embedding extraction protocols.


Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Summary of Revisions, Controlled 19.9x Speedup, and StarEmbed Benchmarks

We thank all three reviewers for their constructive, detailed feedback and their recognition of the extensive empirical evaluation, multi-crop innovations, and downstream utility of READit.

In response to the reviewers' valuable suggestions, we have revised our manuscript (available in the updated PDF). Manuscript Revisions & Text Alignment: Reflecting our new controlled experiments (specifically the hardware-isolated 19.9×19.9\times wall-clock speedup over contrastive baselines and the zero-leakage StarEmbed benchmark), we have updated the Abstract, Rapid Convergence Analysis and Computational Efficiency (Section 4.3), Discussion (Section 6), and Conclusion (Section 7) to strictly align with these empirical findings. All revised passages in the uploaded PDF are highlighted in blue for convenient tracking. Below is an executive summary of our core additions and technical clarifications:

1. Technical Novelty & Controlled Compute Speedup (Appendix F, Table 8): To isolate the self-distillation objective from architectural choices, we executed a strictly controlled ablation on identical hardware (2×2\times NVIDIA T4 GPUs) on 26K-Proxy, fixing the Tiny Transformer backbone (L=4,d=256L=4, d=256), 2 global views, and Cosine scheduler:

  • 19.9×19.9\times Wall-Clock Speedup: Self-distillation reaches its loss basin in 0.380.38 GPU-hours (∼\simEpoch 4) vs. 7.657.65 GPU-hours (∼\simEpoch 123) for contrastive NT-Xent—an empirical 95.0%95.0\% compute reduction.
  • Time-to-Parity: Contrastive learning requires 4−5×4-5\times more compute (∼1.5\sim 1.5h) merely to match the 0.40650.4065 Macro-F1 that READit achieves in 2323 minutes, requiring +6.15+6.15 additional GPU-hours for a marginal +0.028+0.028 gain.
  • Overcoming 2-View Ceilings via Temporal Anchoring: While 2-view distillation converges rapidly, it imposes an information bottleneck. READit introduces Temporal Anchoring (3 deterministic global views spanning start, mid, end + 6 multi-scale local views), raising downstream performance to ∼0.425\sim 0.425 Macro-F1 in just 2.622.62 GPU-hours (∼\simEpoch 8, Figure 11).
  • O(N)\mathcal{O}(N) Linear Memory vs. O(N2)\mathcal{O}(N^2) OOM: Crucially, scaling to 9 views is computationally feasible only under self-distillation due to linear memory scaling O(N)\mathcal{O}(N). An equivalent 9-view contrastive setup triggers immediate Out-of-Memory (OOM) failures due to quadratic O(N2)\mathcal{O}(N^2) negative-pair matrices.

2. Expert-Vetted StarEmbed (ZTF-40K) Benchmark (Appendix E): We evaluated READit on the human-vetted StarEmbed benchmark. Using a 3D spherical kk-d tree matching radius (Δθ≥1.5′′\Delta\theta \ge 1.5''), we isolated a strictly non-overlapping, zero-leakage test set (N=3,487N=3{,}487). READit achieves virtually identical Linear Probing Macro-F1 on this clean split (0.6418±0.00600.6418 \pm 0.0060) compared to the full test set (0.6581±0.00460.6581 \pm 0.0046), ruling out pre-training memorization.

3. Cross-Survey Transfer vs. Pre-Trained Baselines (Section 5.2, Table 4): On simulated LSST data (PLAsTiCC), READit strictly outperforms pre-trained Astra-CLR weights across all regimes:

  • Tier 1 (Domain Shift, 3-Way): 75.31%75.31\% vs. 64.06%64.06\% Recall@1; 60.16%60.16\% vs. 53.59%53.59\% 5-shot Macro-F1.
  • Tier 2 (Comprehensive Transfer with Supernovae, 5-Way): 71.72%71.72\% vs. 61.09%61.09\% Recall@1; 48.73%48.73\% vs. 47.14%47.14\% 5-shot Macro-F1.

4. Dataset Construction & Zero-Leakage Clarification (Appendix A.1): In 120K-Eval, unique physical sources were strictly partitioned into disjoint training (80%) and validation (20%) splits prior to independent oversampling, guaranteeing zero source-level leakage.

5. Downstream Classifier Configurations (Appendix C): We detailed classifier hyperparameters (Linear Probe: lbfgs, C=1.0C=1.0, max_iter=4000; kk-NN: k=20,τ=0.07k=20, \tau=0.07; RF: 200 trees, depth 20) and ASTROMER2/CHRONOS2 embedding extraction protocols.


We believe these comprehensive additions directly resolve the core questions regarding computational efficiency, evaluation rigor, and technical novelty. We provide detailed, point-by-point responses to individual reviewer comments below.

AuthorsOpenReview ↗

General Response Update: StarEmbed Baseline Comparisons (Appendix G, Table 9) & Hyperparameter Sensitivity

In response to follow-up discussion, we have added Appendix G (Table 9) to our revised PDF, benchmarking all baseline models directly on the strictly non-overlapping StarEmbed (ZTF-40K) Test [Filtered] split:

  • Baseline Comparisons (Table 9): READit (0.6418±0.00600.6418 \pm 0.0060 Linear Probe; 0.6262±0.00770.6262 \pm 0.0077 Weighted kk-NN) outperforms all self-supervised baselines—including Chronos-2 (0.63470.6347 LP / 0.58960.5896 kk-NN), ASTROMER-2 (0.57760.5776 LP / 0.57320.5732 kk-NN), and Astra-CLR (0.57750.5775 LP / 0.56560.5656 kk-NN)—and achieves statistical parity with Random Forest trained on domain-expert hand-crafted features (0.6429±0.00890.6429 \pm 0.0089).
  • Classifier Sensitivity: Evaluating across C∈{0.1,1.0,10.0}C \in \{0.1, 1.0, 10.0\} (Linear Probe) and k∈{5,10,20,50}k \in \{5, 10, 20, 50\} (kk-NN) on StarEmbed (Test [Filtered]) confirms that downstream Macro-F1 remains highly stable across hyperparameter scales.

[Update: Astra-CLR values in Table 9 corrected following resolution of an inference checkpoint path].

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.