acceptodds
Under review as a conference paper at ICLR 2027

Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Models

Abstract

Reinforcement learning (RL) is a central approach for improving reasoning capabilities in large language models (LLMs), where training efficiency depends critically on how problems are sampled during optimization. Existing adaptive curriculum learning methods typically prioritize prompts of intermediate difficulty, treating problem selection as a standard bandit problem with independent arms and overlooking the structured, heterogeneous nature of the task space. In this work, we instead frame problem sampling as a manifold-structured bandit problem with endogenous non-stationarity, where relationships between tasks and evolving model capabilities jointly influence learning dynamics. To operationalize this perspective, we introduce **Bayesian Manifold Curriculum (BMC)**, a structure-aware framework that leverages latent representations of LLMs to organize problems into a hierarchical task tree and applies Bayesian learning to guide sampling. Across multiple training setups and evaluation benchmarks, we find that different sampling strategies induce non-trivial tradeoffs between productivity (learning signal), diversity (coverage of the task manifold), and utility (evaluation relevance). These results show that prioritizing difficulty alone is insufficient to ensure strong downstream performance and highlight the importance of incorporating structure and diversity in problem sampling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, large language models, reasoning, multi-armed bandits, Thompson sampling, curriculum learning, Bayesian methods

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BSzhOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper tackles adaptive curriculum learning for LLM reinforcement learning finetuning. It introduces a method that organizes tasks into a hierarchical tree using their latent representations and then introduces a Bayesian batch sampling procedure over this structure. They compare different sampling procedures and find that maximizing learning signal through difficulty is not sufficient to ensure enough training diversity and to maximize downstream performance.

Critical strengths

The paper tackles an important issue in RL finetuning for LLMs, and the construction of a structured latent over training inputs over which sampling can be done is a novel and interesting idea.

Critical weaknesses
  • No external baseline beyond Dynamic Sampling which suffers from high training time due to repeated sampling, this makes it hard to assess the benefit of the method namely the Latent Task Tree construction. The paper cites some alternative works using bandits but doesn't provide any training comparison.
  • The benefit of BMC compared to Tree-only and Difficulty-only ablation is unclear form the results provided by the authors, results on different benchmarks seem close and are provided without any confidence interval.
Questions for authors
  • How do you isolate the benefit of the Latent Task Tree construction itself?
  • Could you provide a training comparison against the bandit-based methods you cite?
2 comments
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for their feedback. We address your comments and concerns below:


Concern 1: External Baselines

An external bandit-based baseline is already included. In Appendix D, we include comparisons to MoPPS [1], which is a representative method for adaptive curriculum learning. We also describe why Difficulty Only is representative of various other adaptive curriculum learning methods that abide by the same difficulty-protocol, which we refer to as the "Standard Bandit Pattern."


Concern 2: Tree Isolation

The Tree Only ablation isolates the role of the Latent Task Tree construction. Selections start from the root of the tree and descend randomly to select problems. When this random selection is replaced with difficulty-based selection, the method becomes BMC. If you take away the tree from BMC, the method becomes Difficulty Only. Tree Only is used throughout the paper.


Concern 3: Benchmark Interpretation

The core contribution of our work is the Productivity-Diversity-Utility decomposition for curriculum learning. We use this decomposition to explain differences between sampling methods that benchmark scores alone may not reveal.

Each axis shows different behavior between the ablations:

Productivity: Tree Only experiences a 40% reduction in effective problems compared to BMC and Difficulty Only (Figure 4). This is because Tree Only prioritizes diverse coverage of problems and ignores difficulty/productivity.

Diversity: BMC achieves approximately 3$\times$ the rarity-weighted exposure of Difficulty Only (Appendix B, Figure 7), indicating broader exposure to problem types. Difficulty Only prioritizes productive difficulty without explicitly accounting for problem type. Tree Only achieves even broader coverage because it prioritizes coverage without accounting for productivity.

Utility: The substantial differences in productivity and diversity do not translate directly into equally large benchmark differences, because the additional problem types encountered may differ in their relevance to the evaluations. BMC-T investigates this distinction: changing which problem types are prioritized improves targeted benchmark scores while maintaining similar productivity (Figure 6). Thus, similar benchmark scores do not imply similar curriculum behavior.


Concern 4: Confidence intervals

Quantifying variability across training seeds requires multiple independent runs per experimental configuration. We provide experiments across different datasets, model sizes, and RL algorithms, but did not repeat each configuration across independent training seeds due to the substantial cost at our scale.


[1] Qu et al., Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?, KDD 2026 (https://arxiv.org/abs/2507.04632)

Thank you for the response. I acknowledge that a comparison against MoPPS is included in Appendix D. However, it mainly uses MoPPS to argue that the Difficulty Only ablation is a representative stand-in for bandit-based curriculum methods. It is not a direct test of whether BMC improves on them.

My main concerns remain. The evidence that BMC outperforms existing methods is limited: against MoPPS, BMC’s only stated advantage on a OOD benchmark is on GPQA-Diamond, and without any confidence intervals or repeated runs, the statistical support for these differences is insufficient. The Tree Only ablation isolates the effect of difficulty-aware selection on the tree, but it does not show that the hierarchical latent structure itself matters compared with alternative structures. Finally, the response acknowledges that the productivity and diversity improvements translate into only modest performance differences, which, given the limited statistical support, leaves the practical benefit of BMC unestablished.

I therefore maintain my score.

Reviewer_QJmpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates adaptive curriculum learning for group-relative reinforcement learning in large language models over a model-induced latent task manifold. Its primary contributions include: (1) constructing Latent Task Trees recursively from intermediate policy representations using local PCA, a chart test, UMAP, and HDBSCAN without external annotations; (2) formulating a hierarchical Bayesian bandit algorithm (BMC) that combines top-down Thompson sampling, surprise-modulated leaf filtering, and bottom-up empirical Bayes propagation; and (3) introducing a utility-aware variant (BMC-T) to analyze trade-offs among productivity, diversity, and downstream utility. The findings are validated primarily on DAPO-Math-17K and AlphaMed19K, though standard BMC does not consistently outperform unstructured bandit baselines on in-distribution benchmarks.

Critical strengths
  1. The conceptual shift from one-dimensional difficulty filtering to a three-axis trade-off (productivity, diversity, and utility) is insightful and practically meaningful. The empirical demonstration that similar training productivity can yield distinct downstream capability profiles (via BMC-T and frontier imbalance analysis) provides a valuable perspective on RLVR data selection.
  2. The Latent Task Tree construction is self-contained and computationally lightweight, requiring only about 1% of total training wall-clock time without relying on external embedding models or human taxonomies. Extensive visualizations across eight diverse datasets confirm that intermediate policy representations naturally induce coherent task hierarchies.
Critical weaknesses
  1. Standard BMC shows weak empirical gains on downstream evaluations compared to simpler baselines: on Qwen3-8B (Figure 5) and AlphaMed19K (Figure 11), it fails to beat the structure-agnostic Difficulty Only baseline across all five math benchmarks and all three medical benchmarks, winning only on OOD GPQA-Diamond. While BMC-T improves math scores, it embeds actual held-out evaluation prompts into the tree to compute target overlap. This transductive use of test distributions prevents a fair inductive comparison against target-blind baselines.
  2. The non-stationary belief update relies on ad-hoc heuristics (Logit-Normal transforms, 3-sigma variance clamping, exponential surprise decay, and logarithmic staleness injection) without formal bandit regret guarantees. Crucially, Appendix K.4 reveals that 80% of prompt beliefs saturate at the maximum uncertainty bound by the end of training. This widespread variance saturation risks degenerating hierarchical Thompson sampling into near-uniform random exploration and likely contributes to the decaying learning signal observed in AlphaMed19K.
Questions for authors

Q1: Can BMC-T achieve consistent downstream gains over Difficulty Only under a strictly inductive protocol where the target distribution is formed by an independent validation set rather than the evaluation test set itself (e.g., using AMC23 to target AIME24/25 without placing AIME prompts in the tree)? This would materially strengthen the practical validity of utility-aware sampling.

Q2: How does the 80% leaf uncertainty saturation reported in Appendix K.4 impact parent variances and sampling behavior in the second half of training, and does reducing staleness injection resolve the learning-signal decay and performance gap on AlphaMed19K? Clarifying whether uncertainty saturation degrades late-stage exploitation is essential for assessing the soundness of the filtering heuristic.

1 comment
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for their feedback. We address your comments and concerns below:


Concern 1: Empirical evidence for BMC and clarifying contributions

The main contribution of the paper is the Productivity-Diversity-Utility decomposition for curriculum learning. We do not claim that BMC is a consistently superior benchmark optimizer. BMC, the Difficulty Only and Tree Only ablations, and BMC-T are introduced in order to investigate how these three dimensions interact.

Not all training problems are equally relevant to a particular benchmark. Thus, a sampling method that is more productive and covers more problem types can still score lower than methods that prioritize problems more relevant to that benchmark. The BMC-T experiments support this explanation: prioritizing problem types related to the targeted benchmarks improves their scores while maintaining similar levels of productivity (Figure 6). Higher benchmark scores therefore need not reflect greater productivity; they can reflect closer alignment between the curriculum and the evaluations. Which curriculum performs better consequently depends on what the experimenter's chosen benchmarks measure. Our contribution is to make these distinctions explicit and provide a framework for investigating and controlling them.


Concern 2: BMC-T Clarification

The target/evaluation separation you describe is already present in the paper. BMC-T (T=AIME2024), shown in green in Figure 6, uses only AIME2024 as its target set. It achieves higher scores than BMC on AMC2023 and AIME2025, neither of which is embedded in the tree. Its performance is similar to Difficulty Only. Thus, the experiment uses AIME2024 as a sort of validation set to guide training, and shows improved performance on evaluation test sets, which is analogous to your proposed example. It shows that utility-aware sampling can improve performance on benchmarks that were not used to construct the curriculum.


Concern 3: Uncertainty Saturation

Earlier iterations of BMC did suffer from unbounded uncertainty growth, which led to the degeneration to near-uniform exploration that you described. The uncertainty cap described in Appendix M.3 was introduced to address this issue. Reaching this cap does not equalize belief means. Candidates with higher estimated learning signal remain favored when their variances are equal.

Parent uncertainty is computed by aggregating child means and uncertainties. The aggregation reduces uncertainty when child means agree, while disagreement can increase uncertainty to prevent overconfident parent beliefs (see Appendix M.2.3). Thus, widespread prompt-level saturation does not force higher-level decisions to become equally uncertain or near-uniform.

We agree that the high saturation rate warrants further investigation. The possible contributors to the 80% saturation rate at the end of training are:

  1. The maximum uncertainty limit is set too low
  2. The rate of uncertainty injection is set too high
  3. Some combination of the above

Adjusting these variables could improve the balance between exploration and exploitation, although we have not tested whether reducing uncertainty injection resolves the AlphaMed-19k decline. We discuss a separate explanation for that decline below.


Concern 4: Learning signal decay on AlphaMed-19k

The AlphaMed-19k results should be interpreted while keeping the available learning signal in mind. Here we used the post-trained Qwen3-8B rather than Qwen3-8B-Base. This model-dataset combination offers fewer productive prompts, as indicated by Dynamic Sampling's increased resampling requirements:

  • With Qwen3-8B-Base on DAPO-Math-17k, Dynamic Sampling requires 1-2 sampling rounds to fill an effective batch, equivalent to generating 2-4 batches with an oversampling ratio of 2.
  • With Qwen3-8B on AlphaMed-19k, it requires 2-3 sampling rounds, equivalent to generating 4-6 batches.

You can view the cumulative batches generated for the different settings here.

Dynamic Sampling does not use BMC's uncertainty mechanism, so this increased sampling requirement provides separate evidence that productive problems are harder to obtain in the medical setting. As fewer prompts provide learning signal, methods that select a batch without repeated rollout filtering, including BMC and Difficulty Only, face a more demanding selection problem.

Uncertainty saturation may also contribute to BMC and Difficulty Only's productivity decay, but the limited supply of productive prompts is an additional factor that must be considered. We view this setting as more of a stress test for adaptive curriculum selection.

Reviewer_ip4qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces "Bayesian Manifold Curriculum" (BMC), a structure-aware curriculum learning framework designed for the reinforcement learning of large language models. The core motivation stems from the observation that existing adaptive sampling methods prioritize learning efficiency while often overlooking the intrinsic relationships between different types of tasks.

BMC employs dimensionality reduction and clustering techniques to construct a "Hierarchical Latent Task Tree" based on the intermediate-layer representations of a policy model. The framework then utilizes Hierarchical Thompson Sampling, non-stationary Bayesian belief updating, and a bottom-up belief propagation mechanism to guide the selection of prompts. Furthermore, the authors propose a variant called BMC-T, which steers the sampling process toward regions associated with the distribution of specific downstream tasks.

Critical strengths

1. Offers meaningful problem formulations and profound insights into curriculum learning.

This paper makes a valuable conceptual contribution to the field of adaptive curriculum learning by explicitly distinguishing between three distinct objectives: "productivity," "diversity," and "downstream utility." This perspective moves beyond traditional difficulty-based sampling methods and highlights the limitations of solely maximizing immediate learning signals, a highly insightful observation that could serve as a useful reference for future research on data selection in RL.

2. Logically coherent framework with a comprehensive empirical analysis.

The proposed framework introduces a novel approach that leverages relationships between training problems through model-derived latent representations and hierarchical Bayesian sampling. The paper not only compares various sampling strategies but also analyzes training efficiency and data coverage.

Critical weaknesses

1. The empirical evidence fails to convincingly demonstrate that this method offers a significant advantage over simpler adaptive sampling approaches.

My primary concern is that the proposed method does not consistently outperform the "Difficulty Only" baseline (which removes the latent task hierarchy while retaining adaptive Thompson sampling). Figure 4 shows comparable training efficiency across these methods, while Figure 5 reveals mixed results regarding downstream performance on mathematical reasoning benchmarks. Although the BMC (Bayesian Meta-Curriculum) approach achieves decent results on the GPQA-Diamond dataset, the gains are relatively modest; furthermore, it is unclear whether these results are statistically robust across multiple independent training runs. Additionally, in the medical domain experiments, both the "Difficulty Only" and "Dynamic Sampling" methods actually demonstrate superior downstream performance.

2. Fails to adequately disentangle and validate the individual contributions of the "latent task geometry" versus the "hierarchical Bayesian mechanism."

While the paper demonstrates that the learned task hierarchy captures intrinsic relationships, going beyond what would be expected from random partitioning, the explanatory power of this structure itself appears limited. In Appendix M.7, for the 8B model scale, the average explained variance (R²) of the learning signals at the prompt level is only 0.00875. This raises a question: to what extent does the identified geometric structure actually correspond to the learned relational associations?

Furthermore, existing ablation studies fail to clearly disentangle whether the observed performance gains stem from the learned latent geometric structure and hierarchical belief propagation, or simply from a more general, diversity-aware reweighting mechanism.

Questions for authors

Same as weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for their feedback. We address your comments and concerns below:


Concern 1: Empirical evidence for BMC and clarifying contributions

The main contribution of the paper is the Productivity-Diversity-Utility decomposition for curriculum learning. We do not claim that BMC is a consistently superior benchmark optimizer. BMC, the Difficulty Only and Tree Only ablations, and BMC-T are introduced in order to investigate how these three dimensions interact.

Not all training problems are equally relevant to a particular benchmark. Thus, a sampling method that is more productive and covers more problem types can still score lower than methods that prioritize problems more relevant to that benchmark. The BMC-T experiments support this explanation: prioritizing problem types related to the targeted benchmarks improves their scores while maintaining similar levels of productivity (Figure 6). Higher benchmark scores therefore need not reflect greater productivity; they can reflect closer alignment between the curriculum and the evaluations. Which curriculum performs better consequently depends on what the experimenter's chosen benchmarks measure. Our contribution is to make these distinctions explicit and provide a framework for investigating and controlling them.


Concern 2: Addressing simpler diversity mechanisms

We do not claim that simpler diversity mechanisms cannot approximate sampling with a Latent Task Tree. Our argument is that defining diversity in curriculum learning requires considering two issues:

  • Human-defined data categorizations do not necessarily correspond to how the data is internally organized by the policy. What is considered diverse to a human may not be diverse to the policy.
  • Diversity can exist at multiple levels of detail (e.g. breaking Math into Algebra and Geometry, and Geometry into 2D and 3D geometry). A single flat partition cannot represent these nested relationships. Latent Task Trees strictly generalize flat clustering methods and can produce flat clusters as shown in Appendix F.

The Latent Task Tree reduces reliance on human-defined categories and allows for hierarchical decompositions when such complexity is detected. Human-defined categories or flat clustering may produce similar sampling behavior in particular settings. This does not eliminate the need to consider policy-dependent organization and multiple levels of detail when defining diversity.


Concern 3: Disentangling geometry from hierarchical Bayesian selection

The desired disentanglement between the latent task geometry and hierarchical Bayesian selection mechanism is already present in the paper.

First, the Tree Only ablation uses the geometry/hierarchy without the hierarchical Bayesian mechanism. Samples start from the root and descend the tree randomly. BMC replaces this with hierarchical Thompson sampling informed by adaptive Bayesian belief updates. This comparison examines what adaptive Bayesian sampling adds to the same latent hierarchy.

Second, the Diversity section (Appendix B) reports the Structure Gain for the Difficulty Only ablation. Difficulty Only does not use the latent task tree for sampling. Yet, the Structure Gain over time suggests that the tree still partially explains variation in estimated learning signal even though it was never used for hierarchical sampling. This shows that the tree’s explanatory power is not merely an artifact of using that same tree to guide sampling.


Concern 4: Interpreting low R2R^2

The reported R2R^2 measures alignment with running learning-signal estimates, rather than fresh measurements of every prompt at every training step, which would require prohibitively expensive repeated rollouts across the full dataset. Structure Gain contextualizes the small absolute value by asking how much more variation the Latent Task Tree explains than expected by chance. The results indicate modest absolute explanatory power but greater alignment than random groupings. That this also holds for Difficulty Only, which does not use the tree for sampling, further supports non-random alignment between policy behavior and latent geometry.

Our work does not claim that the tree explains most variation in difficulty or fully characterizes the policy's learned associations. Alternative latent coordinate systems may capture these relationships better, and our work can serve as inspiration for such investigations. More precisely quantifying these relationships remains a fruitful direction for future work.


Concern 5: Independent training runs

We provide experiments across different datasets, model sizes, and RL algorithms, but did not repeat each configuration across independent training seeds due to the substantial cost at our scale.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.