acceptodds
Under review as a conference paper at ICLR 2027

Continual Visual and Verbal Learning Through a Child's Egocentric Input

Abstract

Children learn the meanings of words from a continuous, temporally structured stream of egocentric experience. Recent work shows that neural networks can also learn word-referent mappings from a child's egocentric video recordings, but they cycle through the shuffled data for hundreds of epochs, contrasting with how children actually encounter their environment. We introduce BabyCL, a continual multimodal framework that combines streaming visual representation learning with an image–text contrastive objective to learn in a single chronological pass from naturalistic child-headcam video–language streams. BabyCL combines a multi-stage temporal segmentation of the stream with a dual replay buffer that independently manages visual and multimodal histories, and it jointly optimizes three objectives on a shared backbone. Against a streaming baseline that shares BabyCL's replay, schedule, and labeled-pair exposure but omits its visual objectives, BabyCL's frozen visual features yield consistently higher linear-probe accuracy, both in domain and on a separate child-headcam corpus. BabyCL also improves four-alternative forced-choice word–referent accuracy over this baseline and narrows the gap to offline training. Together, these results show that meaningful word-referent mappings can emerge under training conditions much closer to a child's actual experience.

Primary area:
applications to neuroscience & cognitive science
Keywords:
continual learning, multimodal learning, egocentric learning, vision-language learning, developmental learning, representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uxK1OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BabyCL, a multimodal representation learning framework that learns from chronologically ordered video-language streams captured from a developing child's egocentric perspective. Unlike conventional offline training with shuffled i.i.d. data over multiple epochs, BabyCL processes the input stream in a single chronological pass. Built upon a replay-based continual learning framework, BabyCL augments the existing cross-modal contrastive objective with additional visual self-supervised losses, demonstrating improvements over its streaming baseline across a diverse range of downstream tasks, including vision-language alignment and visual classification, while narrowing the performance gap with offline training.

Critical strengths
  1. Meaningful research problem. The paper addresses an interesting and meaningful research problem by doing mutimodal representation learning like a developing child who process chronologically ordered visual streams. This setting provides a biologically inspired perspective on visual and language representation learning.

  2. Comprehensive experimental evaluation. The paper includes extensive experiments and ablation studies, along with a diverse set of downstream evaluations. They cover vision-language alignment (4AFC), zero-shot generalization (VTWT and Baby Winoground), visual representation quality (in-domain and out-of-domain linear probing), as well as retrieval and localization tasks. Such a broad evaluation provides a comprehensive assessment of the learned representations across different capabilities.

Critical weaknesses
  1. The writing could be improved. The paper lacks a clear overview of the entire training pipeline, making it difficult to understand how the different components (eg. stream processing, temporal segmentation, buffer updates) fit together. Moreover, the computation matching between baselines is also somewhat confusing, as different comparisons match different quantities (e.g., labeled-pair exposure versus gradient steps). I suggest providing a overview of the complete training pipeline and adding a comparison table that explicitly lists the batch size, number of samples per step, total gradient steps, and cumulative sample exposure for each compared model.

  2. The memory buffer size is relatively large. A smaller memory buffer would improve the scalability of the proposed framework.

Questions for authors
  1. If I understand correctly, there is no guarantee that every frame is used for model updates at least once. I suggest reporting how the timestamp distribution of sampled training frames evolves over the course of training, including 1) whether the average sampled timestamp increases as the stream progresses and 2) the temporal gap between the current stream timestamp and the sampled frames. This analysis would help clarify to what extent the effective training data distribution preserves the chronological progression of the input stream.

  2. Regarding the temporal classification loss LS\mathcal{L}_S, if class labels are assigned locally within each batch, their meanings may not be consistent across batches. For instance, the same event may receive different labels in different batches, while the same label may correspond to entirely different events across batches. If these labels are used to train a shared classification head, the inconsistent supervision could make it difficult for the model to learn meaningful temporal representations. Could the authors clarify how the classification loss is implemented? It would also be helpful to report the classification accuracy. If the accuracy is close to chance level, this could potentially explain the limited contribution of LS\mathcal{L}_S observed in the ablation study.

3 comments

Thank you for your thoughtful review, and in particular for the analysis of the temporal classification loss.

The paper lacks a clear overview of the entire training pipeline [...] Moreover, the computation matching between baselines is also somewhat confusing, as different comparisons match different quantities.

We will improve the writing of the method and experiment sections according to your suggestion.

One full step is as follows. Every 12 seconds of the stream, the online segmenter re-segments within short-term memory. A finished event segment enters the short-term tier of the visual buffer, and the utterance–frame pairs inside it enter the short-term tier of the multimodal buffer. After each new event is inserted, k=8 updates are made, each drawing 3 events of 32 frames from the visual buffer and 32 pairs from the multimodal buffer and computing the three losses on the shared backbone. The matching rule is: streaming models are matched in gradient steps, so their multimodal-pair exposure is also equal; streaming and offline models are matched in multimodal-pair exposure, and their gradient steps differ because the offline batch holds 8 pairs.

Table-A4. Training budget of each compared model. "multimodal-pair exposure" = steps × multimodal pairs per step.

ModelStream orderGradient stepsSamples per stepmultimodal-pair exposureReplay buffers
BabyCL (k=8)chronological26,56896 visual frames + 32 multimodal pairs850Kvisual + multimodal
CL-CVCL (k=8)chronological26,56832 multimodal pairs850Kmultimodal
Offline CVCL (compute-matched)i.i.d., ~25 epochs106K8 pairs850Knone
Offline CVCL (full)i.i.d., 400 epochs1.69M8 pairs13.5Mnone

The memory buffer size is relatively large.

The language supervision in the child video datasets are very scarce (the whole SAYCam stream only contains 33,737 utterance–frame pairs), and therefore it's difficult to get good multimodal performance with a small replay buffer (in such case we will only be taking very few gradient steps can many word concepts, insufficient for effective learning).

For the visual buffer, we are able to shrink it by quite a bit though, given that we use a larger batch size of 512 (4× the original setting). From Table A3 below we show that the visual buffer can be reduced to only 10% of default (i.e. 1% of seen for short buffer, and 5% of seen for long bugger) and both the visual and multimodal performance can be fully retained.

Table-A3. Visual-buffer size ablation at batch size 512 over three seeds per setting. Buffer capacities are percentages of their respective defaults.

Visual / multimodal bufferLabeled-S linear probeLabeled-S 4AFCBabyView 4AFC
100% / 100%45.64 ± 0.6239.37 ± 0.8331.37 ± 1.80
20% / 100%46.93 ± 0.8439.73 ± 2.1229.82 ± 1.14
10% / 100%46.64 ± 0.6940.25 ± 0.5930.22 ± 1.37

I suggest reporting how the timestamp distribution of sampled training frames evolves over the course of training, including 1) whether the average sampled timestamp increases as the stream progresses and 2) the temporal gap between the current stream timestamp and the sampled frames.

By the end of training 82.9% of visual events have been sampled at least once, each record about 21 times on average; 98.6% of utterance–frame pairs have been used at least once, about 30 times on average.

We report the other requested information in Table-A5 below.

Table-A5. Training statistics at five different progress checkpoints (unit is hours).

Training progressStream positionMean sampled visual data timevisual data temporal gap median [p10, p90]Mean sampled multimodal data timemultimodal data temporal gap median [p10, p90]
5%23.01.920.7 [19.3, 23.7]10.313.1 [3.5, 20.3]
25%53.817.636.1 [21.7, 50.4]28.024.7 [6.7, 46.1]
50%83.936.445.9 [20.9, 76.3]32.652.4 [18.3, 75.3]
75%118.253.262.6 [26.4, 107.3]59.367.7 [18.8, 105.4]
100%145.670.173.6 [23.7, 129.8]59.395.2 [46.2, 132.8]

Overall the effective training distribution keeps the forward progression of the stream. The temporal gap grows over training, but this is by design of the buffer strategy so that it still revisit earlier events to prevent forgetting.

Regarding the temporal classification loss, if class labels are assigned locally within each batch, their meanings may not be consistent across batches. [...] Could the authors clarify how the classification loss is implemented? It would also be helpful to report the classification accuracy. If the accuracy is close to chance level, this could potentially explain the limited contribution of LS\mathcal{L}_S observed in the ablation study.

Thank you for your valuable advice. The classification accuracy of LS\mathcal{L}_S as implemented in the submission is indeed at chance.

As a result, we plan to replace LS\mathcal{L}_S with a supervised contrastive loss, and we show encouraging results below. In the supervised contrastive objective, frames of the same event are positives, frames of the other events negatives; the MLP is modified to an 128-d projection head; the temperature used is 0.1.

Table-A6. Original LS\mathcal{L}_S implementation vs. the supervised contrastive loss on all metrics, mean ± 95% CI over 5 seeds.

MetricOriginal LS\mathcal{L}_S implementationSupervised contrastive for LS\mathcal{L}_SΔ
Labeled-S 4AFC42.72 ± 1.3643.55 ± 1.01+0.83
Labeled-S linear probe44.98 ± 0.0647.83 ± 0.11+2.85
mini-ImageNet linear probe25.26 ± 0.3427.23 ± 0.31+1.98
VTWT59.63 ± 1.8360.64 ± 1.51+1.01
Baby Winoground (overall)13.81 ± 1.2315.01 ± 1.64+1.21
BabyView 4AFC33.18 ± 0.7533.60 ± 1.67+0.41
BabyView linear probe50.83 ± 0.4752.40 ± 1.21+1.57

We see that the supervised contrastive objective improves the performance of all evaluation metrics. Given these encouraging results, we plan to change the definition of LS\mathcal{L}_S in the paper to the supervised contrastive loss and rerun the BabyCL experiments. The paper's main conclusions remain the same (BabyCL still outperforms the baselines).

Thank you for your reply. I suggest that this branch should be replaced by contrastive loss, where the classification loss is not correct.

Reviewer_m58xOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BabyCL, a continual multimodal learning framework designed to learn visual representations and word-referent associations from children's egocentric experiences. Unlike earlier work that trains many epochs on shuffled child headcam data, BabyCL processes the original video stream in temporal order. Evaluations on SAYCam, BabyView, and other benchmarks show improvements over a matched streaming baseline. However, the gains in word-referent learning are smaller, and offline training remains stronger.

Critical strengths
  • This is an important and developmentally motivated learning setting. The authors address a central limitation of previous child headcam models: their reliance on shuffled, multi-epoch training. They show that meaningful word-referent learning is possible when the original data stream is only seen once chronologically.
  • This work also shows strong improvements in visual representation learning and cross-corpus transfer. The benefits of BabyCL are especially strong for frozen visual representations; there is strong linear probe accuracy gains on mini-ImageNet and BabyView. This suggests that the further visual supervision helps the model learn more transferable representations.
Critical weaknesses

However, I feel there are a couple of important weaknesses:

  1. The method iteslf still relies on substantial replay, which limits its developmental interp. Although BabyCL processes the original video stream chronologically, individual training examples are revisited many times through reply. Performance also appears to considerably lower when replay capacity is reduced. This shows that the authors' findings show learning under a chronological streaming constraint, but don't necessarily establish that learning is possible under the more restrictive memory and experience limitations that apply to human children.

  2. The word-grounding improvements are quite modest and there is limited evidence for the importance of temporal structure, as the primary improvement over the matched streaming baseline is less than 3 percentage points. Most of the improvement relative to naive streaming seems to come from the replay-update budget. Also, the authors emphasize chronological learning, but do not establish that chronological order itself is beneficial compared with a shuffled stream under the same replay and compute constraints. Further, the model performs relatively poorly when trained directly on BabyView, showing that the approach may be somewhat dataset-specific. In general, the paper shows that the proposed BabyCL works, but evidence for generalization is much weaker.

Questions for authors
  1. Can you please report statistical significance and confidence intervals of more of the primary results?
  2. How does BabyCL perform when the stream order is randomized while keeping other conditions and constraints equivalent?
  3. What is the minimum replay capacity needed to preserve BabyCL's advantage over the memory-matched streaming baseline?

Also, I believe there may be a couple works which are worth reading and potentially citing, since they seem relevant to the motivation and BabyView experiments in this paper. For example, Feng et al. (2024) (https://aclanthology.org/2024.emnlp-main.1231) looks at how developmental ordering affects learning from child-directed speech, and Feng et al. (2026) (https://arxiv.org/abs/2610.36679) looks at how differences across children's language environments (using BabyView transcripts) affect learning.

2 comments

Thank you for your thoughtful review.

The method itself still relies on substantial replay, which limits its developmental interp.

We agree with this limitation. The paper does not claim full developmental interpretability, and we are certainly not claiming that the proposed method can directly apply to human children, though we try to simulate the chronological single pass with bounded memory setting. We will clarify these in the revised paper.

The word-grounding improvements are quite modest and there is limited evidence for the importance of temporal structure.

While we acknowledge that the word-grounding improvements are quite modest, our methods utilizes all the visual frames without utterance text supervision, which CVCL discards. This allows us to significantly outperform CVCL methods in terms of the learned visual representation, as shown in Table A1a below.

Table-A1a. BabyCL vs. CL-CVCL (k=8) on three visual representation benchmarks (linear probing on image classification) over 5 seeds, mean ± 95% CI; Δ is the difference of means and p a Welch t-test.

MetricBabyCLCL-CVCL (k=8)Δp
Labeled-S linear probe44.98 ± 0.0640.20 ± 0.25+4.78<0.001
BabyView linear probe50.83 ± 0.4735.67 ± 0.68+15.16<0.001
mini-ImageNet linear probe25.26 ± 0.3413.75 ± 0.36+11.51<0.001

We will include this result in the revised paper and clarify the contributions.

The model performs relatively poorly when trained directly on BabyView, showing that the approach may be somewhat dataset-specific.

In terms of the absolute performance, training from scratch on BabyView is indeed worse than on SAYCam. However, the proposed method still comfortably outperform the baselines on BabyView, suggesting that the gains of the method is not dataset-specific.

Can you please report statistical significance and confidence intervals of more of the primary results?

In the submission only Labeled-S 4AFC in the main table had a CI; the other metrics were single-seed. We will replace every table in the main text with 5-seed statistics. Table-A1b below provides mean ± 95% CI over 5 seeds with Welch t-tests on the primary metrics.

Table-A1b. BabyCL vs. CL-CVCL (k=8) on four multimodal benchmarks used in the paper, over 5 seeds, mean ± 95% CI; Δ is the difference of means and p a Welch t-test. Some numbers here are slightly different from the original paper due to seed variance.

MetricBabyCLCL-CVCL (k=8)Δp
Labeled-S 4AFC42.72 ± 1.3640.72 ± 1.49+2.000.09
BabyView 4AFC33.18 ± 0.7529.65 ± 0.97+3.53<0.001
VTWT59.63 ± 1.8358.84 ± 1.12+0.790.50
Baby Winoground (overall)13.81 ± 1.2314.08 ± 1.61−0.270.80

How does BabyCL perform when the stream order is randomized while keeping other conditions and constraints equivalent?

In Table-A2 below we compare the performance of BabyCL on chronological stream vs. shuffled stream. Note that we shuffle the arrival of the event segments instead of the raw frames so that the utterances are still attached to a contiguous set of frames and our online segmentation module can utilize these boundaries.

We observe that the performance on most benchmarks are slightly higher in the shuffled setting, but the chronological setting is not far behind. We don't claim in the paper that chronological learning is beneficial over shuffled learning. We emphasize chronological multimodal learning since it's a somewhat novel training paradigm and also seens as a challenge by community wisdom.

Table-A2. Chronological vs. shuffled stream, mean ± 95% CI over 5 seeds.

SettingLabeled-S 4AFCLabeled-S linear probemini-ImageNet probeBabyView 4AFCBabyView linearVTWTWinoground
BabyCL, chronological (default)42.72 ± 1.3644.98 ± 0.0625.26 ± 0.3433.18 ± 0.7550.83 ± 0.4759.63 ± 1.8313.81 ± 1.23
BabyCL, event order shuffled43.55 ± 0.7846.08 ± 0.4126.23 ± 0.6332.69 ± 1.2052.52 ± 1.3860.23 ± 2.4915.23 ± 0.50
CL-CVCL (k=8), chronological40.72 ± 1.4940.20 ± 0.2513.75 ± 0.3629.65 ± 0.9735.67 ± 0.6858.84 ± 1.1214.08 ± 1.61

What is the minimum replay capacity needed to preserve BabyCL's advantage over the memory-matched streaming baseline?

The language supervision in the child video datasets is very scarce (the whole SAYCam stream only contains 33,737 utterance–frame pairs), and therefore it's difficult to get good multimodal performance with a small replay buffer (in such case we will only be taking very few gradient steps can many word concepts, insufficient for effective learning).

For the visual buffer, we are able to shrink it by quite a bit though, given that we use a larger batch size of 512 (4× the original setting). From Table A3 below we show that the visual buffer can be reduced to only 10% of default (i.e. 1% of seen for short buffer, and 5% of seen for long bugger) and both the visual and multimodal performance can be fully retained.

Table-A3. Visual-buffer size ablation at batch size 512 over three seeds per setting. Buffer capacities are percentages of their respective defaults.

Visual / multimodal bufferLabeled-S linear probeLabeled-S 4AFCBabyView 4AFC
100% / 100%45.64 ± 0.6239.37 ± 0.8331.37 ± 1.80
20% / 100%46.93 ± 0.8439.73 ± 2.1229.82 ± 1.14
10% / 100%46.64 ± 0.6940.25 ± 0.5930.22 ± 1.37

Also, I believe there may be a couple works which are worth reading and potentially citing

Thank you for the references. We will cite them in the revised paper.

Reviewer_uHbiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BabyCL, a continual multimodal learning framework that learns word–object associations from children's egocentric video and language streams in a single chronological pass. It combines temporal segmentation, dual replay buffers, and joint visual and image–text contrastive learning to better reflect how children learn from experience. Experiments show that BabyCL improves visual representations and word–referent recognition compared to a streaming baseline, while narrowing the performance gap with offline training.

Critical strengths
  • The paper addresses an interesting and important problem: how AI models can learn word–object associations from continuous, real-world experiences, rather than relying on shuffled datasets and repeated training over many epochs. This is particularly relevant to understanding how children learn language and to developing more realistic continual learning systems.
  • The proposed idea is intuitive and easy to follow, with a clear motivation and a straightforward overall framework.
Critical weaknesses
  • The main weakness is the limited technical novelty. BabyCL largely combines two existing approaches: the streaming visual representation learning framework (Yang & Ren), including hierarchical temporal segmentation and replay buffers, and the image–text contrastive learning objective (Vong et al) . While the integration is reasonable, the paper does not introduce a substantially new learning mechanism or architecture.
  • It is also unclear what technical challenges arise from combining these existing methods and whether the proposed integration offers insights beyond the existing approaches.
  • The paper shows quite limited comparison with other approaches in the experiments.
  • The paper tested only on children's learning videos; I'm not asking for more experiments, yet why can't we test the method on other videos for continual learning tasks?
  • Some parts of the writing, particularly the mathematical formulation, appear slightly unnatural. For example, in Equation (1), I assume that 0.5 is used as the weight for all three loss terms. If so, the parentheses around the first two terms seem unnecessary and could be removed for clarity?
Questions for authors

I have two major questions: the novelty of the proposed method when combining several previous works, and the experiment is limited to a very specific children's video domain.

1 comment

Thank you for your thoughtful review.

The main weakness is the limited technical novelty.

We believe the main contribution of this work is not a new loss or architecture, but making dense video self-supervision and sparse utterance supervision train together on one chronological stream and one backbone, and separating the role of each part with compute-matched controls.

We also want to note that many implementation details are different from Yang & Ren due to the additional text supervision used in this work. In BabyCL, the online segmentation module is guided by the start and end times of the utterances. Each form of supervision (visual and multimodal) has its own two-tier buffers.

We will clarify these contributions in the revised paper.

It is also unclear what technical challenges arise from combining these existing methods and whether the proposed integration offers insights beyond the existing approaches.

The difficulty is that the two forms of supervision differ widely in time scale and density. Video arrives at 5 frames per second, continuous and highly redundant; utterance–frame pairs arrive every few tens of seconds on average, sparse and noisy. Placed in one buffer with one capacity and one eviction policy, either the video crowds out the language samples or the language samples are replayed far too often. Giving each form of supervision its own pair of buffers, each with its own capacity, lets us set their capacities independently.

Training the two together gives findings that neither method alone provides. First, most of the streaming gain comes from the number of replay updates per observation: CL-CVCL goes from 31.48 at k=1 to 40.72 at k=8, and the visual objectives add two to three points on 4AFC on top (Table 1 of the submission). Second, what the visual objectives mainly add is transferable visual representation (Table-A1 below). Against compute-matched CL-CVCL, the mini-ImageNet linear probe is higher by 11.51, the BabyView linear probe by 15.16 and BabyView 4AFC by 3.53, all significant; CVCL, with only the cross-modal objective, does not have this property.

Table-A1. BabyCL vs. CL-CVCL (k=8) over 5 seeds on three visual representation evaluation benchmarks (linear probing on image classification), mean ± 95% CI; Δ is the difference of means. The two differ only in the visual objectives.

MetricBabyCLCL-CVCL (k=8)Δ
Labeled-S linear probe44.98 ± 0.0640.20 ± 0.25+4.78
BabyView linear probe50.83 ± 0.4735.67 ± 0.68+15.16
mini-ImageNet linear probe25.26 ± 0.3413.75 ± 0.36+11.51

The paper shows quite limited comparison with other approaches in the experiments.

CL-CVCL is the only streaming baseline because it is the only one that can be matched item by item. It shares the stream, the segmentation, the buffers, the batch composition, the optimizer and the number of steps with BabyCL, and only sets the visual loss coefficients to zero, so the difference is attributable to the visual objectives alone. Adding k=1 (same stream and buffers, one update per observation) and the two offline CVCL runs (no replay, i.i.d. over the whole corpus, one matched in updates and one trained for the full 400 epochs) covers the range from streaming with few updates to i.i.d. multi-epoch training. Other continual-learning methods (MinRed, ER-style replay) are designed for purely visual or task-bounded settings without sparse paired language; porting them would first require a multimodal version of each, and the differences that introduces cannot be controlled.

The paper tested only on children's learning videos; I'm not asking for more experiments, yet why can't we test the method on other videos for continual learning tasks?

Our method requires a timestamped video stream and text aligned to it. The usual continual-learning video benchmarks (Kinetics subsets, CORe50-style) have no natural language aligned to the video, so the cross-modal objective cannot be used; narrated egocentric data such as Ego4D narrations is a feasible next step that we will leave to future work.

Some parts of the writing, particularly the mathematical formulation, appear slightly unnatural. For example, in Equation (1), I assume that 0.5 is used as the weight for all three loss terms. If so, the parentheses around the first two terms seem unnecessary and could be removed for clarity?

The parentheses were meant to express the balance between these two groups of training objectives: the visual objective and the multimodal objective. However we do agree that the parentheses are unnecessary and could be removed for clarity. We will correct these issues and improve the writing of the mathematical formulation in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.