- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes BabyCL, a multimodal representation learning framework that learns from chronologically ordered video-language streams captured from a developing child's egocentric perspective. Unlike conventional offline training with shuffled i.i.d. data over multiple epochs, BabyCL processes the input stream in a single chronological pass. Built upon a replay-based continual learning framework, BabyCL augments the existing cross-modal contrastive objective with additional visual self-supervised losses, demonstrating improvements over its streaming baseline across a diverse range of downstream tasks, including vision-language alignment and visual classification, while narrowing the performance gap with offline training.
Critical strengths
-
Meaningful research problem. The paper addresses an interesting and meaningful research problem by doing mutimodal representation learning like a developing child who process chronologically ordered visual streams. This setting provides a biologically inspired perspective on visual and language representation learning.
-
Comprehensive experimental evaluation. The paper includes extensive experiments and ablation studies, along with a diverse set of downstream evaluations. They cover vision-language alignment (4AFC), zero-shot generalization (VTWT and Baby Winoground), visual representation quality (in-domain and out-of-domain linear probing), as well as retrieval and localization tasks. Such a broad evaluation provides a comprehensive assessment of the learned representations across different capabilities.
Critical weaknesses
-
The writing could be improved. The paper lacks a clear overview of the entire training pipeline, making it difficult to understand how the different components (eg. stream processing, temporal segmentation, buffer updates) fit together. Moreover, the computation matching between baselines is also somewhat confusing, as different comparisons match different quantities (e.g., labeled-pair exposure versus gradient steps). I suggest providing a overview of the complete training pipeline and adding a comparison table that explicitly lists the batch size, number of samples per step, total gradient steps, and cumulative sample exposure for each compared model.
-
The memory buffer size is relatively large. A smaller memory buffer would improve the scalability of the proposed framework.
Questions for authors
-
If I understand correctly, there is no guarantee that every frame is used for model updates at least once. I suggest reporting how the timestamp distribution of sampled training frames evolves over the course of training, including 1) whether the average sampled timestamp increases as the stream progresses and 2) the temporal gap between the current stream timestamp and the sampled frames. This analysis would help clarify to what extent the effective training data distribution preserves the chronological progression of the input stream.
-
Regarding the temporal classification loss , if class labels are assigned locally within each batch, their meanings may not be consistent across batches. For instance, the same event may receive different labels in different batches, while the same label may correspond to entirely different events across batches. If these labels are used to train a shared classification head, the inconsistent supervision could make it difficult for the model to learn meaningful temporal representations. Could the authors clarify how the classification loss is implemented? It would also be helpful to report the classification accuracy. If the accuracy is close to chance level, this could potentially explain the limited contribution of observed in the ablation study.
3 comments
Regarding the temporal classification loss, if class labels are assigned locally within each batch, their meanings may not be consistent across batches. [...] Could the authors clarify how the classification loss is implemented? It would also be helpful to report the classification accuracy. If the accuracy is close to chance level, this could potentially explain the limited contribution of observed in the ablation study.
Thank you for your valuable advice. The classification accuracy of as implemented in the submission is indeed at chance.
As a result, we plan to replace with a supervised contrastive loss, and we show encouraging results below. In the supervised contrastive objective, frames of the same event are positives, frames of the other events negatives; the MLP is modified to an 128-d projection head; the temperature used is 0.1.
Table-A6. Original implementation vs. the supervised contrastive loss on all metrics, mean ± 95% CI over 5 seeds.
| Metric | Original implementation | Supervised contrastive for | Δ |
|---|---|---|---|
| Labeled-S 4AFC | 42.72 ± 1.36 | 43.55 ± 1.01 | +0.83 |
| Labeled-S linear probe | 44.98 ± 0.06 | 47.83 ± 0.11 | +2.85 |
| mini-ImageNet linear probe | 25.26 ± 0.34 | 27.23 ± 0.31 | +1.98 |
| VTWT | 59.63 ± 1.83 | 60.64 ± 1.51 | +1.01 |
| Baby Winoground (overall) | 13.81 ± 1.23 | 15.01 ± 1.64 | +1.21 |
| BabyView 4AFC | 33.18 ± 0.75 | 33.60 ± 1.67 | +0.41 |
| BabyView linear probe | 50.83 ± 0.47 | 52.40 ± 1.21 | +1.57 |
We see that the supervised contrastive objective improves the performance of all evaluation metrics. Given these encouraging results, we plan to change the definition of in the paper to the supervised contrastive loss and rerun the BabyCL experiments. The paper's main conclusions remain the same (BabyCL still outperforms the baselines).
Thank you for your reply. I suggest that this branch should be replaced by contrastive loss, where the classification loss is not correct.
Thank you for your thoughtful review, and in particular for the analysis of the temporal classification loss.
We will improve the writing of the method and experiment sections according to your suggestion.
One full step is as follows. Every 12 seconds of the stream, the online segmenter re-segments within short-term memory. A finished event segment enters the short-term tier of the visual buffer, and the utterance–frame pairs inside it enter the short-term tier of the multimodal buffer. After each new event is inserted, k=8 updates are made, each drawing 3 events of 32 frames from the visual buffer and 32 pairs from the multimodal buffer and computing the three losses on the shared backbone. The matching rule is: streaming models are matched in gradient steps, so their multimodal-pair exposure is also equal; streaming and offline models are matched in multimodal-pair exposure, and their gradient steps differ because the offline batch holds 8 pairs.
Table-A4. Training budget of each compared model. "multimodal-pair exposure" = steps × multimodal pairs per step.
The language supervision in the child video datasets are very scarce (the whole SAYCam stream only contains 33,737 utterance–frame pairs), and therefore it's difficult to get good multimodal performance with a small replay buffer (in such case we will only be taking very few gradient steps can many word concepts, insufficient for effective learning).
For the visual buffer, we are able to shrink it by quite a bit though, given that we use a larger batch size of 512 (4× the original setting). From Table A3 below we show that the visual buffer can be reduced to only 10% of default (i.e. 1% of seen for short buffer, and 5% of seen for long bugger) and both the visual and multimodal performance can be fully retained.
Table-A3. Visual-buffer size ablation at batch size 512 over three seeds per setting. Buffer capacities are percentages of their respective defaults.
By the end of training 82.9% of visual events have been sampled at least once, each record about 21 times on average; 98.6% of utterance–frame pairs have been used at least once, about 30 times on average.
We report the other requested information in Table-A5 below.
Table-A5. Training statistics at five different progress checkpoints (unit is hours).
Overall the effective training distribution keeps the forward progression of the stream. The temporal gap grows over training, but this is by design of the buffer strategy so that it still revisit earlier events to prevent forgetting.