acceptodds
Under review as a conference paper at ICLR 2027

Less Data, Smaller Batches: When Implicit Regularization Helps Informed Meta-Learning

Abstract

Informed meta-learning predicts on a new task using its few observed examples and task-specific knowledge. When the training pool contains only a few tasks, a model may learn spurious correlations between that knowledge and the observations that fail under distribution shift. Small example batches can discourage reliance on such shortcuts in supervised learning; we ask whether small task batches help when entire tasks are scarce. A finite-pool analysis shows that, for vanilla SGD, smaller task batches increase a local penalty on disagreement among task gradients. We analyze why Adam does not inherit the same scalar result, then evaluate both optimizers. With Adam, the small-versus-full-batch predictive log-likelihood benefit is much larger with 25 training tasks than with 200, on both controlled sinusoids and a prospective benchmark constructed from measured weather. This supports a scarcity-dependent optimization effect; the experiments do not identify a particular shortcut. Separately, pre-materializing tasks and overlapping data transfer yield an approximately end-to-end pipeline speedup on one GPU. Code: https://github.com/thirdperson11111/iclr_38898.gitgithub.com/thirdperson11111/iclr_38898.git.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
meta-learning, spurious correlation, group robustness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pjtnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how task batch size affects the generalization performance of informed meta-learning when only a limited number of training tasks are available. The authors analyze the implicit regularization effect of small task batches under SGD and conduct experiments using Adam and SGD. Results on synthetic sinusoidal tasks and real-world weather data suggest that smaller batches can improve predictive log-likelihood. However, the theoretical novelty is limited, and the experimental evaluation lacks sufficient depth and rigor. Overall, the paper reads more like a preliminary exploratory study than a mature and well-substantiated research contribution.

Critical strengths
  1. The paper studies an interesting and practical problem: improving meta-learning performance when training tasks are limited.
  2. The authors provide a mathematical analysis of task-gradient covariance and conduct several experiments to investigate the effects of batch size and task scarcity.
Critical weaknesses

Critical Weaknesses

1. Limited theoretical novelty and a gap between theory and experiments.

The theoretical contribution is relatively limited, as the main analysis largely extends existing SGD implicit regularization results to task-level gradients. The authors should better clarify the novelty compared with previous studies. More importantly, the theoretical results only apply to vanilla SGD, while the main experimental improvements are obtained using Adam. This mismatch leaves the connection between the theoretical analysis and the observed performance gains unclear.

2. Insufficient experimental validation and lack of direct evidence for the proposed mechanism.

Although smaller task batches improve predictive log-likelihood in several settings, the experiments do not directly demonstrate that these improvements result from reduced shortcut learning. Moreover, the evaluation mainly focuses on the INP framework and a limited number of benchmarks, with insufficient comparisons against other meta-learning models and regularization methods. Some important details of the real-world experiments are also missing, raising concerns about reproducibility. More comprehensive experiments and stronger baselines are needed to support the main claims.

Questions for authors
  1. Can the authors explain how the SGD-based theoretical analysis supports the much larger improvements observed with Adam?
  2. Is there direct experimental evidence that smaller task batches reduce shortcut learning rather than simply changing the optimization dynamics?
  3. Do the main conclusions remain valid for intermediate task-pool sizes and other meta-learning architectures?
Reviewer_X5WMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper is about how task diversity affects the implicit bias of batch size in meta learning. They show on two empirical settings that with low task diversity, small batch size results in better OOD error than large batch size, but that this advantage shrinks with higher task diversity.

Critical strengths

Questions about implicit bias in training are important, and this paper doesn't limit itself to a single setting when studying it, using both a synthetic setting and a setting based on real data.

Critical weaknesses

The paper is very difficult to read, lacks a coherent story and is mostly describing and justifying the methods/empirical results, without good motivation for the wider interpretation of the results in the context of the field. The experiments are also insufficient to support the broad claims the paper wants to make, testing only two or three batch sizes per setting, and letting the larger batch size be much larger in the more diverse settings. More analysis, like line graphs showing different batch sizes or levels of diversity would be much more compelling than comparing two points per setting. Overall, the paper reads more like an llm written report on a series of experiments than a research paper.

Questions for authors

Were learning rate sweeps performed? They are mentioned, but the appendix just indicates a single lr was used mostly and the others tested were unstable.

Were the number of optimizer steps fixed, or the total number of samples seen? Ie, did batch size 5 and 200 run for the same number of steps, or was batch size 5 rand for 40 times more steps? It appears the number of steps was fixed, if thats true, would it change the results if we ran the smaller batch sizes for longer?

What was done to limit or prevent over fitting? Did you sweep weight decay values?

Reviewer_tqA1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether informed meta-learning benefits from smaller task batches when the number of training tasks is very limited. In supervised learning smaller batches can suppress shortcut learning so the authors ask if a similar effect exists in meta-learning under task scarcity. The paper provides a finite-pool analysis showing that vanilla SGD with smaller task batches increases a local penalty on task gradient disagreement, and explains why Adam does not inherit this scalar result. Experiments with Adam show that the log-likelihood benefit of small over full batches is significantly larger at 25 tasks than at 200 tasks using both controlled sinusoid tasks and a self-built weather benchmark. This supports the claim of scarcity-dependent optimization effects. The paper also reports approximately 15x end-to-end pipeline speedup from task pre-materialization and data transfer overlap.

Critical strengths
  1. The finding is practically valuable since smaller batches perform better under task scarcity providing direct guidance for resource-constrained meta-learning.
  2. The paper combines theory and empiricism. The finite-pool analysis yields a modified SGD objective with a task-gradient covariance term and the authors honestly acknowledge that Adam does not inherit the scalar SGD result.
Critical weaknesses
  1. The theory only covers local finite-pool settings for vanilla SGD while the main experiments are conducted with Adam. This creates a weak theory-experiment connection and there is no positive theory for Adam.
  2. The core benefit primarily appears in log-likelihood yet on sinusoid tasks the LL improvement comes with worse RMSE raising questions about the practical value of the main metric gain. And the benchmark is insufficient, which limits both its validity and reproducibility. Furthermore, the exploration of "scarcity" is confined to two extreme scenarios, 25 tasks versus 200 tasks, without a characterization of the critical transition points between them.
Questions for authors

When LL improves but RMSE worsens which metric should take priority?

Comments on the paper

1 comment

Thank you for the reviews. Informed meta-learning uses a new task’s few observations and task-specific knowledge to make predictions. Scarce training tasks can make associations between these inputs look reliable even when they fail under shift. We ask how the benefit of small task batches changes with the number of training tasks.

Task scarcity. With Adam, small batches improve out-of-distribution (OOD) predictive log-likelihood (LL) more at 25 tasks than at 200 in both benchmarks. Each 25-task pool is nested in its corresponding 200-task pool. We average two initializations per pool and report paired pool-level differences with 95% confidence intervals.

BenchmarkSmall − full at 25Small − full at 200Difference in gains
Sinusoids+69.70 [46.08, 93.32]+2.41 [−6.26, 11.07]+67.29 [46.40, 88.19]
Prospective weather+42.85 [28.94, 56.76]+18.00 [10.83, 25.17]+24.85 [5.22, 44.47]

The gain was larger at 25 tasks in all eight sinusoid and all five weather pools. At 200 tasks, the sinusoid gain is unresolved and the weather gain remains positive. As X5WM notes, full batch grows with the pool. The result estimates the gain from switching away from each pool’s full batch. A common batch-size grid at intermediate pool sizes would help distinguish the effects of pool size and batch size. A transition point and the effect of task diversity at fixed pool size remain open.

Training budget and tuning. The main comparisons matched optimizer updates (4,000 per sinusoid run and 2,000 per weather run). Sinusoid tasks were reshuffled without replacement and partial final batches were retained. Weather batches of 5 and 200 saw the same number of updates, with 40 times as many task exposures at B = 200. In a separate 25-task sinusoid control, matching 25,000 exposures gave 5,000/4,000/1,000 updates at B = 5/8/25. At the final checkpoints, LL relative to full batch was +51.2 [22.9, 79.6] for B = 8 and −29.1 [−45.8, −12.3] for B = 5. The effect of longer weather runs remains open.

At 25 sinusoid tasks, the Adam learning-rate scaling control gave +83.9 [60.0, 107.7] LL for B = 8 versus full batch. Accumulating microbatches of eight into an effective full batch gave −2.6 [−14.4, 9.2] versus native full batch. We ran a vanilla-SGD learning-rate sweep and selected the weather learning rate on validation data. We used held-out validation for checkpoint selection in the main sinusoid runs and evaluated all configurations on the same fixed test tasks. A full Adam learning-rate grid and a weight-decay sweep remain open.

SGD, Adam, and shortcuts. We apply the random-reshuffling analysis of Smith et al. (2021) to gradients of the expected per-task training loss, including the Informed Neural Process KL term. For vanilla SGD with equal batches B dividing a fixed pool N and a sufficiently small constant step η, the local one-pass objective weights the trace of task-gradient covariance by η(N−B)/(4BN), using the 1/(N−1) covariance convention. This term vanishes at full batch and grows as B decreases. If a shortcut increases task-gradient covariance, this term penalizes the shortcut-dependent solution more at smaller B. On sinusoids, SGD at B = 5 and η = 0.001 gave +1.33 [0.68, 1.99] LL versus full batch. Adam’s moments depend on gradient history, so the SGD result gives no prediction for the sign or size of Adam’s gains. Shortcut suppression remains a hypothesis. We did not intervene on a known shortcut, the normalized gradient-noise diagnostic was weakly associated with OOD LL, and a likelihood gain also appeared on in-distribution sinusoids.

LL versus RMSE. LL evaluates the predictive distribution. RMSE takes priority when the application uses point forecasts and squared error reflects the cost. At 25 sinusoid tasks, B = 8 improved LL and raised empirical coverage of nominal 90% intervals from 63.4% to 76.2%, with wider intervals and worse RMSE (+0.046 [0.029, 0.063]). At 25 weather tasks, B = 5 improved LL, reduced RMSE from 2.569 to 2.157, and increased coverage from 73.0% to 83.7% with narrower intervals.

Weather protocol and scope. Each task is one day of measured temperatures. We use 12 first-half measurements to predict all 144 second-half measurements under seasonal shift. Knowledge is restricted to calendar information, prior-day summaries, and information available by the first-half cutoff. A standard Neural Process also showed a larger small-batch LL gain at 25 than at 200 tasks (+28.42 [16.93, 39.90] for the difference). Generalization beyond this model family and regression remains open. We would lead with the paired pool-size result and consolidate station and split dates, feature construction, pool IDs and seeds, and tuning and checkpoint rules. We also welcome clarification of tqA1’s ethics flag so we can address the specific concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.