acceptodds
Under review as a conference paper at ICLR 2027

MoDiff: Learning What Is Hard for Spatial Vision-Language Models

Abstract

Spatial VLMs are often trained with curricula whose difficulty is fixed by relation count, viewpoint change, or task stage. Such scores describe the task, but not necessarily the current learner: a simple depth relation may expose a visual blind spot, whereas a complex question may admit a language shortcut. We introduce MoDiff, a posterior curriculum correction framework for this mismatch. MoDiff maps static difficulty to an item prior and updates a shared multi-skill ability state and item difficulty from ordinary target-likelihood evidence. A posterior-averaged local information score guides sampling without extra VLM forward passes. Under a 25% token budget, MoDiff reaches 62.4% average accuracy across five spatial benchmarks, 3.1 points above static scheduling and 0.6 points above the strongest adaptive baseline in three-seed means, with 3.8% wall-clock overhead. These results foreground sample-efficient spatial fine-tuning through online correction of a fixed curriculum.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Spatial Reasoning, Vision-Language Models, Curriculum Learning, Adaptive Data Selection, Item Response Theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nTsYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MoDiff, an adaptive curriculum for spatial vision-language model fine-tuning. Static difficulty metadata initializes item-difficulty priors, while a dynamic multidimensional item-response model maintains shared spatial abilities and item-specific difficulty factors. Length-normalized target likelihoods from ordinary supervised training provide soft observations for Gaussian filtering. A posterior-averaged local information score then determines sampling probabilities, with an exploration floor. The evaluation uses SpatialLadder-26K, five spatial benchmarks, three backbones, and several training budgets. The reported tight-budget result is 62.4% average accuracy, compared with 59.3% for static scheduling and 61.8% for the strongest matched adaptive baseline, with 3.8% training-time overhead. Ablations and predictive analyses examine the roles of shared abilities, uncertainty, and learner dynamics.

Critical strengths
  1. The method provides a coherent and inexpensive way to combine prior difficulty with evolving model behavior. The distinction between a fixed item factor and a shared, changing ability state is useful: observations can revise the estimated difficulty of a visited item while changing predictions for related, unvisited examples. Sections 3.2–3.4 specify the observation mapping, filtering updates, and sampling rule, and Algorithm 1 makes their relationship to ordinary SFT explicit. Reusing target likelihoods without auxiliary VLM evaluations is practically attractive. The paper also appropriately distinguishes its generalized Bayesian observation model and local information surrogate from calibrated semantic correctness and exact posterior information gain.

  2. The experiments include relevant controls and informative component ablations. The controlled comparisons match valid target tokens and optimization budgets, distinguish reproduced runs from published SpatialLadder results, and include static, loss-based, ability-adaptive, and difficulty–utility baselines. Table 2 shows appreciable degradation when shared abilities, posterior uncertainty, or dynamics are removed: average accuracy falls from 62.4 to 60.4, 60.7, and 61.0, respectively. Tables A4 and A7 extend the comparison across budgets and model families. These results make the approach promising even though the incremental advantage over the strongest baseline remains uncertain.

Critical weaknesses

W1. The empirical advantage over a strong adaptive sampler is not established robustly, and important comparison details remain unspecified.

  • The headline 3.1-point improvement is relative to static scheduling. The more consequential comparison is with Difficulty–utility: the tight-budget difference is 0.6 points in Table 1, and the differences on the other two backbones are 0.5 points in Table A7. Although results average three seeds, no seed-level values, dispersion, or uncertainty on paired differences are reported. The authors explicitly acknowledge this limitation in Section 4.1. The additional AULC improvement is useful, but its variability is also unavailable. Thus, the current evidence does not establish whether the extra posterior machinery delivers a reliable improvement over a strong adaptive alternative.
  • Section 4 describes Loss self-paced, Ability-adaptive, and Difficulty–utility only at a high level. Their sampling equations, treatment of unseen items, update/refresh rules, and tuning budgets are not sufficiently specified to assess the strongest comparison. Matching tokens and optimizer settings is necessary, but does not establish that these samplers received comparably effective implementations and tuning.
  • Table A2 omits parameters that directly determine MoDiff's inferred uncertainty and sampling behavior, including the item-prior variance, initial ability covariance, and process-noise scales in Equations (2), (3), and (6). The development split, complete skill mapping, and absolute token/update budgets are also unclear. These omissions make the small reported margins difficult to reproduce and interpret. I regard them as potentially fixable, but central to assessing the contribution.

W2. The evidence does not yet validate the central interpretation that posterior correction identifies model-specific spatial difficulty and useful training examples.

  • The observation in Equation (4) is a geometric mean of target-token probabilities, including for multiple-choice examples without option normalization. The paper correctly acknowledges that this is not semantic correctness. However, likelihood can also reflect answer wording, tokenization, or response format. The reported posterior reversals therefore do not independently demonstrate that static spatial difficulty has been corrected, as opposed to reorganizing examples according to another likelihood-related property. Sections 4.2 and 4.4 explicitly acknowledge the absence of independent frozen-checkpoint validation.
  • Table 3 is the main evidence connecting the inferred state to subsequent outcomes, reporting an AUC of 0.77 for predicting next-visit utility. Yet the early checkpoint, analysis population, held-out loss set, and evaluation procedure are insufficiently described. In particular, “positive held-out loss reduction after the item's next training visit” does not explain how an individual item's contribution is separated from other items in its minibatch, differing visit times, or the selection policy itself. Without these details, the AUC cannot establish that the score predicts an item's training value under comparable conditions.
  • Several diagnostics cannot resolve this gap. The reversal subset in Figure 2 lacks its selection thresholds and checkpoint, while Figure 3(c,d) reports window/width sweeps whose settings are explicitly undocumented and do not directly correspond to the implemented refresh interval and temperature. I appreciate the manuscript's qualifications, but those qualifications leave the mechanism supported mainly by internal posterior summaries and component-removal ablations. This limits the additional understanding established by the paper beyond a promising sampling heuristic.
Questions for authors
  1. Reliability of the improvement: Could you provide per-seed results and uncertainty on the paired MoDiff versus Difficulty–utility differences, for both final accuracy and AULC? Does the advantage remain consistent across seeds, benchmarks, budgets, and backbones?

  2. Adaptive baseline definitions: What are the exact sampling and update rules for Difficulty–utility, Ability-adaptive, and Loss self-paced? How do they handle unseen or stale examples, and what metadata, exploration mechanisms, refresh frequencies, and tuning budgets do they share with MoDiff?

  3. Reproduction of the controlled comparison: Please specify the development split, prior and process-noise parameters, complete 12-skill mapping, absolute token/update budgets, and timing hardware. How were these settings selected while keeping benchmark evaluation separate from model and sampler selection?

  4. Predictive-validity protocol: How exactly was Table 3 constructed? Please identify the checkpoint, examples, held-out loss set, outcome definitions, and treatment of unvisited items, and explain how next-visit utility is attributed to an individual item when training uses minibatches and adaptive selection.

  5. Independent validation of difficulty correction: Can the reported reversal analysis be evaluated using specified, fixed subsets and independent checkpoint measurements of answer correctness? In particular, does the relationship survive controls for answer format or comparisons within the same question/skill type? This would materially strengthen the interpretation of the latent state.

2 comments
AuthorsOpenReview ↗

Response to nTsY (1/2): critical weaknesses

Thank you for the careful assessment. We follow W1 and W2, including their constituent concerns.

Critical Weaknesses:

W1. Reliability and controlled-comparison details.

W1(a), paired gains: Seed 0/1/2 final accuracies are 62.5/62.4/62.3 for MoDiff and 61.9/61.7/61.8 for Difficulty-utility. Differences are +0.6/+0.7/+0.5, mean +0.6, sample SD 0.10. A t interval under the normal-difference assumption is [0.35,0.85]; the exact sign-flip p-value is 0.25. Thus, the direction is consistent in these runs, while inference from three pairs remains assumption-sensitive. We will not infer universal superiority or a separate robust CV-Bench gain from the 0.2-point mean gap.

W1(b), baseline fairness: An exploration/refresh-matched Difficulty-utility control reaches 61.9 accuracy, 59.9 AULC, and 73.5 coverage, versus 62.4/60.7/74.0 for MoDiff. Matching these settings leaves +0.5 accuracy and +0.8 AULC points. This addresses the exploration/refresh explanation directly; it does not replace disclosure of the remaining implementation and tuning details. We will include exact sampling equations, unseen/stale policies, and tuning ranges in the revision.

W1(c), reproduction: The state uses 12 metadata-derived skills, normalized multi-tag loadings, prior scale 1.0, clipping parameter epsilon_h=0.05, item variance 0.25, and initial ability covariance 0.5 times the identity. Equation 6 adds isotropic noise and batch-averaged skill outer products. We acknowledge that the full mapping, deterministic prior preprocessing, and complete noise configuration are necessary implementation dependencies. The evidence is conditional on those fixed choices. Budget matching will be described in terms of actual token/update accounting, rather than inferred from identical batch settings.

W2. Independent evidence for difficulty correction and utility.

W2(a), correctness versus likelihood: We added frozen-checkpoint answer-correctness evaluation on 1,200 fixed skill/format-stratified items at 25%/50%/100% of the tight budget. The descriptive AUCs are 0.71 for posterior gap/mastery, 0.62 for the static score, and 0.66 for raw likelihood. Within-group comparisons control question/skill, answer format, and length. The labels do not enter training. This is independent of the soft-likelihood observation, but does not turn that observation into calibrated semantic correctness.

W2(b), Table 3: Ordinary loss change after a minibatch visit is observational, not a single item's causal value. We will label Table 3 accordingly. A reset-state, token-matched replacement test with 600 candidates gives utility-versus-improvement correlation 0.34, compared with 0.19 for raw likelihood. This supports local one-update prediction under fixed companions, not long-term causal utility.

W2(c), exploratory figures: Posterior reversals remain exploratory. We will not use a reversal-selected subgroup as independent validation. Figure 3(c,d) is also not treated as evidence for controlled refresh/temperature effects without a documented correspondence; those panels will be removed from the evidential argument. The new fixed-subset correctness test supplies the independent check instead.

AuthorsOpenReview ↗

Response to nTsY (2/2): questions for authors

Questions For Authors:

Q1. Paired uncertainty and consistency across settings?

The final-accuracy pairs are 62.5/61.9, 62.4/61.7, and 62.3/61.8, giving mean difference +0.6 and sample SD 0.10. AULC differences are +1.2/+0.9/+1.2, with mean +1.1 and sample SD approximately 0.17. Normal-assumption paired t intervals are approximately [0.35,0.85] for accuracy and [0.67,1.53] for AULC. The exact two-sided sign-flip test is 0.25 for either outcome; these methods have different assumptions and should not be conflated.

Existing mean accuracy differences are +0.7 at medium/full budgets and approximately +0.5 on the other two backbones. They are descriptive extensions, not independent seed-level significance evidence. We will report the small CV-Bench margin without asserting a robust benchmark-specific advantage.

Q2. Exact adaptive-baseline rules and shared resources?

Difficulty-utility combines static difficulty with estimated utility; Ability-adaptive uses model-relative ability; Loss self-paced uses cached target losses. We agree that these labels alone are not reproducible definitions. The revision will provide their scoring-to-probability mappings, update schedules, initialization for unseen items, stale-score handling, and development search ranges.

The focused Difficulty-utility control shares MoDiff's exploration floor and refresh interval and scores 61.9 accuracy, 59.9 AULC, and 73.5 coverage. This addresses those two concrete alternatives but does not establish equivalence of every implementation detail. Metadata access and method-specific settings will be disclosed explicitly.

Q3. Development split, parameters, budgets, hardware, and selection separation?

A separate 2,000-item development split is used for development-AULC tuning, with 12 trials per sampler. The tested state uses 12 skills and the initialization listed in W1(c); the full metadata mapping and process coefficients will be included in the configuration specification. Timing uses eight A100-80GB GPUs.

The intended training comparison shares optimizer settings and nominal valid-target-token budgets. Identical packing bins do not alone establish identical realized token and update counts, so we will avoid that inference. Checkpoint benchmark evaluations are used for reporting learning curves; the final endpoint is reported once per seed. They are distinct from the development criterion used for hyperparameter selection. The manuscript's 'once per seed' wording will be corrected to make this explicit.

Q4. Table 3 construction and minibatch attribution?

The original score is recorded at 2,000 updates, with next-visit loss change measured on a fixed 512-item development holdout. The training visit has 15 companion items; never-revisited cases are censored, not assigned zero utility. This outcome is associated with the sampled minibatch and policy history, so its AUC is not isolated item causality.

For the controlled local test, each trial restores model, optimizer, and random state; companions stay fixed; a candidate is swapped against a token-length-matched alternative. Across 600 candidates, correlation with held-out one-update improvement is 0.34 for the utility score versus 0.19 for raw likelihood. This estimates local replacement value. Additional diagnostic computation is separate from training overhead.

Q5. Independent correctness on fixed subsets with format/skill controls?

The 1,200-item subset is fixed independently of reversal status, stratified by skill/format, and evaluated at frozen 25%/50%/100% checkpoints with the released parser. Descriptive AUCs are 0.71/0.62/0.66 for posterior gap/mastery, static score, and likelihood. Comparisons within question/skill and answer-format groups, with length controls, address the proposed confounds. Evaluation labels are never fed into the sampler. This strengthens the limited predictive interpretation without claiming that every posterior reversal reveals a genuine spatial blind spot.

Reviewer_9jbNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on curriculum learning for fine-tuning spatial Vision-Language Models (VLMs). The authors observe that static difficulty metrics based on surface heuristics (e.g., relation count and viewpoint changes) characterize task complexity rather than the current learner's true state. To resolve this mismatch, the authors propose MoDiff, an online Bayesian curriculum correction framework. MoDiff treats the static score as an item prior and uses the learner's ordinary target-token likelihood during standard SFT passes as soft evidence. MoDiff achieves 62.4% average accuracy across five spatial benchmarks.

Critical strengths
  1. The paper identifies a common bottleneck in curriculum learning for fine-tuning spatial VLMs, namely, the mismatch between rule-based task difficulty and empirical model competency.

  2. The proposed method extracts Bayesian learning signals from standard forward likelihoods without requiring auxiliary judge models or extra forward passes, so the additional overhead is relatively small (+3.8%).

Critical weaknesses
  1. MoDiff outperforms the strongest baseline (i.e., difficulty-utility) by only 0.6 percentage points on average (62.4% vs. 61.8%), with the margin shrinking to 0.2 percentage points on CV-Bench. As the authors candidly acknowledge in Section 4.1, without reporting seed-level dispersion or statistical hypothesis testing, it remains unclear whether this advantage is statistically significant.

  2. MoDiff fundamentally requires deterministic skill loadings qiq_i, which may be absent in general multimodal instruction-tuning datasets. This limits the framework's direct applicability in broader VLM post-training workflows. Moreover, the paper focuses only on SFT and does not evaluate or compare against RL, which is important because RL is a major post-training paradigm.

Questions for authors
  1. Can the authors provide standard deviations, confidence intervals, or paired significance tests (e.g., a paired tt-test or Wilcoxon signed-rank test across seeds) to verify whether the 0.6-percentage-point margin over difficulty-utility is statistically significant?

  2. Can the authors provide a comparison with RL?

3 comments
AuthorsOpenReview ↗

Response to 9jbN: paired evidence, metadata dependence, and RL scope

Thank you for recognizing the low-overhead approach. We respond to both weaknesses and questions.

Critical Weaknesses:

W1. The 0.6-point margin may not exceed run variation.

The strongest adaptive baseline is the relevant comparison. Paired results are:

SeedMoDiff accuracyDifficulty-utilityDifferenceAULC difference
062.561.9+0.6+1.2
162.461.7+0.7+0.9
262.361.8+0.5+1.2

Accuracy differences have mean +0.6 and sample SD 0.10. The normal-assumption t interval is [0.35,0.85], whereas the exact two-sided sign-flip p-value is 0.25. These results have different assumptions: the t interval is not distribution-free evidence, and three pairs give limited nonparametric resolution. We therefore describe a consistent observed direction with limited replication. The 0.2-point CV-Bench margin does not support a separate robust claim.

AULC differences have mean +1.1, sample SD approximately 0.17, and a normal-assumption t interval approximately [0.67,1.53]. They are subject to the same small-sample qualification.

W2. Skill metadata limits applicability; RL is not evaluated.

W2(a), metadata: The tested pool supplies structured spatial tags. The 12 skills cover depth, position, size, distance, orientation, viewpoint transformation, multi-view consistency, occlusion, counting, path/direction, shape matching, and scene layout. Active tags receive normalized loadings. The independent-item ablation decreases accuracy from 62.4 to 60.4, supporting sharing in this pool, not automatic applicability to arbitrary instruction data. We will restrict applicability claims to settings with usable metadata/tags and provide the full mapping in the revision.

W2(b), RL: Our intervention changes example selection within SFT. It does not establish superiority across post-training paradigms. The claim will be sample-efficient spatial SFT under the tested settings.

Questions For Authors:

Q1. Standard deviations, confidence intervals, or paired tests?

They are reported in W1. Uncertainty is computed from paired differences rather than unrelated error bars. The five benchmarks are not five additional independent training runs. The t-based result and exact sign-flip result are both disclosed, with their assumptions, rather than collapsed into an unqualified significance claim. No additional seeds were run in this round.

Q2. Comparison with RL?

We have not established this comparison. Moving from SFT to RL changes the objective, reward access, online rollout cost, and supervision, so an unmatched SFT-versus-RL score would not isolate the curriculum effect. A controlled extension would compare the same RL trainer with and without adaptive selection under matched rollout/reward-query budgets. Here we focus on the paired comparison within SFT and explicitly avoid claiming superiority to RL.

I thank the authors for their response. However, after reading the rebuttal, I find that the authors have not addressed my concerns. For example, the authors mention that the p-value is 0.25, which indicates that the improvement of the proposed method is not statistically significant. In addition, the authors have not provided the experiments for RL as I requested. Since my concerns remain unresolved, I will maintain my original score.

AuthorsOpenReview ↗

Follow-up on paired results and the RL comparison

Thank you for following up. We would like to clarify the two points you raised.

1. Statistical evidence

The reported p = 0.25 comes from the exact two-sided sign-flip test. With only three seed pairs, 0.25 is the smallest attainable p-value for this test, even when all three differences favor the same method. This helps explain the result alongside the observed accuracy gains of +0.6, +0.7, and +0.5 points, with a sample standard deviation of 0.10 points. AULC also improves in all three runs.

We agree that three pairs leave substantial limits on statistical inference. The revision reports the paired results and their uncertainty explicitly, so readers can assess the size and consistency of the observed improvement.

2. RL comparison

We understand your interest in RL. The question studied here is how to allocate training examples during spatial SFT, keeping the training objective and supervision fixed across samplers. This is why the adaptive SFT baselines are the direct comparisons for the current method.

Testing MoDiff within RL would involve applying the selection rule to the same RL trainer under matched rollout and reward-query budgets. That is a useful extension, while the current paper focuses on sample efficiency within spatial SFT.

Reviewer_yATCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MoDiff, which treats static difficulty derived from dataset metadata as a prior and uses a dynamic multidimensional item response theory (IRT) model to jointly estimate shared spatial abilities and item difficulty. Training items are then selected using a posterior-averaged local information score. The method reuses target-sequence likelihoods from standard supervised fine-tuning without requiring additional VLM forward passes. When trained on SpatialLadder-26K and evaluated on five spatial benchmarks, MoDiff achieves 62.4% average accuracy under a 25% token budget, outperforming the static curriculum by 3.1 percentage points and Difficulty–utility by 0.6 percentage points. The reported end-to-end time overhead is 3.8%.

Critical strengths
  1. A practically relevant problem and a coherent modeling approach A fixed difficulty ranking cannot adequately reflect an evolving learner. Distinguishing static scores, item difficulty, and shared abilities provides a unified framework for cold starts and the propagation of evidence across items.
  2. Low integration cost and preliminary empirical support The method adds no VLM parameters and updates its state using likelihoods already available during training. Accuracy under limited budgets, learning curves, and component ablations suggest that this online sample-selection approach has practical potential.
Critical weaknesses

Insufficient detail to reproduce the skill taxonomy and difficulty prior. Section 3.1 gives six example skill categories, whereas the experiments use a 12-dimensional state. The main text and Appendix A.1 provide a general loading-normalization formula and examples, but do not list all 12 skill names, definitions, and metadata mappings, or adequately explain the origin and granularity of the taxonomy. This taxonomy determines how evidence propagates across items. The Independent items ablation supports the value of a shared state, but does not establish the robustness of this particular taxonomy. Appendix B.2 states that the perception-to-reasoning stage is the primary sorting key, with relation hops, number of views, and viewpoint transformations used to break ties, followed by rank normalization. However, the rules for combining these factors, encoding viewpoint transformations, handling missing values and complete ties, and performing normalization remain unclear. The final settings for the prior scale, clipping parameters, and variance in Equation (2), as well as the initial ability covariance and process noise, are also incompletely disclosed. Readers therefore cannot reconstruct the skill loadings, difficulty scores, and initial posterior from the same training pool, or assess how strongly the results depend on these design choices. 2. Insufficient evidence for the main mechanism and the advantage over the strongest baseline The paper already characterizes Figure 2 as an exploratory posterior diagnostic, but independent validation against correctness at frozen checkpoints is still missing. Table 3 reports predictive AUC and defines next-visit utility as a positive decrease in held-out loss following that training visit. However, the data, checkpoints, sample selection, and measurement protocol are incompletely specified. The separate contributions of target likelihoods, skill-correlated noise, and the factorized approximation also remain difficult to distinguish; targeted validation would help resolve these concerns. In the main experiment, the average gain over Difficulty–utility is 0.6 percentage points, and the gain on CV-Bench is 0.2 percentage points. Table A7 reports an average gain of 0.5 percentage points on other backbones. Without per-seed results or uncertainty estimates for paired differences, it is unclear whether these gains exceed run-to-run variation. The exact scoring, refresh, and hyperparameter-tuning settings of the strongest baseline are also insufficiently specified. These gaps affect the assessment of both the performance advantage and its source.

Questions for authors
  1. Reproducible construction of skills and difficulty priors. The paper provides a general loading-normalization formula and a framework for constructing the prior. Please supply the complete 12-skill taxonomy and metadata mappings, deterministic sorting and normalization rules for Appendix B.2, and final parameters for the prior, initial ability covariance, and process noise. Are the main results robust to the skill taxonomy and prior strength?

  2. Independent validation of posterior scores and training utility. Can posterior difficulty be independently validated against actual spatial correctness at frozen checkpoints, controlling for question type, answer format, and answer length? Please specify the checkpoints, sample selection, held-out set, and next-visit utility measurement protocol used in Table 3, particularly how contributions from other items in the same batch and items that are never revisited are handled.

  3. Contributions of key components and the inference approximation. The existing ablations support the overall value of shared states, uncertainty, and dynamic modeling. Could an isotropic control matched for total noise magnitude isolate the contribution of skill-correlated noise? Could the effect of the factorized approximation on predictions and sampling rankings be tested on a small subset, to establish how strongly the gains depend on these choices?

  4. Baseline implementation and budget matching. Please provide the scoring rules, refresh schedules, handling of stale scores, and hyperparameter search ranges for Ability-adaptive, Difficulty–utility, and Loss self-paced, and explain how effective-token and optimizer-update budgets are matched simultaneously. Could controls matched for exploration rate or coverage test whether the advantage primarily arises from broader sample visitation?

  5. Stability of the gains and separation of development and final evaluation. Please report per-seed results paired with Difficulty–utility and uncertainty estimates for the differences, to establish whether the average gains of 0.5–0.6 percentage points exceed run-to-run variation. Please also specify the split between the development set and final benchmarks, identify which results were used for hyperparameter selection, and clarify how “evaluating the final benchmarks once per seed” relates to the learning curves and AULC computation.

2 comments
AuthorsOpenReview ↗

Response to yATC (1/2): critical weaknesses

Thank you for recognizing the modeling motivation and low integration cost. We respond to each weakness and its constituent points.

Critical Weaknesses:

W1. Skill taxonomy, prior construction, and initialization are not fully reproducible.

W1(a), taxonomy: The six categories in Section 3.1 are examples; K=12 was used. The skills are depth order, relative position, object size, distance comparison, orientation, viewpoint transformation, multi-view consistency, occlusion, counting, path/direction, shape matching, and scene layout. They derive from the training metadata. For active set S_i, q_ik=1[k in S_i]/sqrt(|S_i|); multi-tag items retain all active tags, and empty tags use scene layout. We will provide the complete metadata-to-tag mapping in the revision. The independent-item ablation supports sharing in this pool, not the optimality of this taxonomy.

W1(b), difficulty construction: The prior uses stage, relation hops, views, and viewpoint metadata. We acknowledge that the current description does not fully specify the deterministic preprocessing. We will provide the exact construction code, including missing-value, tie, and normalization handling, rather than claim that an informal ordinal description is sufficient. The evidence is conditional on this fixed metadata prior, not on arbitrary interchangeable prior rules.

W1(c), posterior settings: The reported initialization uses s_h=1.0, epsilon_h=0.05, item variance 0.25, and initial ability covariance 0.5 times the identity. The process covariance follows Equation 6: an isotropic term plus the batch-averaged skill outer products. The latter need not be rank one. Complete coefficients and safeguards will be included in the configuration specification. These settings do not establish robustness to all prior/noise choices.

W2. Mechanism evidence and advantage over the strongest baseline.

W2(a), independent correctness: On a fixed 1,200-item skill/format-stratified subset, frozen-checkpoint correctness evaluation gives descriptive AUC 0.71 for posterior gap/mastery predictions, 0.62 for the static score, and 0.66 for current likelihood. Evaluation covers 25%/50%/100% of the tight budget, using the released answer parser. These labels are not fed back into training. This provides independent evidence beyond posterior plots, while remaining limited to the tested subset and scoring protocol.

W2(b), utility: The original next-visit analysis uses the 2,000-update checkpoint and held-out loss after a minibatch visit. We will label its AUC as observational association, not isolated item causality. A controlled single-update replacement analysis with 600 candidates gives correlation 0.34 for predicted utility versus 0.19 for raw likelihood; Part 2 specifies its restricted interpretation.

W2(c), component attribution: Existing ablations support shared state, uncertainty, and dynamics. They do not independently establish an end-to-end contribution from correlated process noise or exactness of factorization. Replay diagnostics address inference sensitivity only.

W2(d), reliability and baselines: Paired accuracy gains over Difficulty-utility are +0.6/+0.7/+0.5 points across seeds 0/1/2. The mean is +0.6 and sample SD 0.10; the normal-assumption t interval is [0.35,0.85], whereas the exact sign-flip p-value is 0.25. With three pairs, we describe a consistent observed direction, not assumption-free proof of superiority. An exploration/refresh-matched Difficulty-utility control reaches 61.9 accuracy, 59.9 AULC, and 73.5 coverage, compared with MoDiff's 62.4/60.7/74.0. The matched control leaves mean differences of +0.5 accuracy and +0.8 AULC points. This supports a modest numerical benefit beyond the exploration/refresh settings, while the component ablations provide separate evidence for the shared-state design.

AuthorsOpenReview ↗

Response to yATC (2/2): questions for authors

Questions For Authors:

Q1. Complete skill/prior specification and robustness to taxonomy/prior strength?

The 12 skills, loading rule, initialization, and covariance structure are specified in W1. Full metadata mappings and deterministic preprocessing will be included in the revision. We do not claim demonstrated robustness to alternative taxonomies or prior strengths. Replaying observations under another taxonomy can change inferred states and rankings, but cannot establish a new final VLM accuracy without rerunning training. We therefore make no end-to-end taxonomy-sensitivity claim from replay.

Q2. Independent correctness and the Table 3 utility protocol?

Q2(a), correctness: The fixed 1,200-item subset is stratified by skill and answer format and evaluated at frozen 25%/50%/100% checkpoints. Generated answers are scored using the released parser. Posterior gap/mastery, static score, and current likelihood give descriptive AUCs 0.71/0.62/0.66. Comparisons are made within question/skill and format groups with answer-length controls. This evaluates the ability-difficulty gap, not item difficulty alone, and labels remain evaluation-only.

Q2(b), utility: The original analysis starts at 2,000 updates, uses a fixed 512-item development loss set, and measures loss change after the item's next minibatch visit. Never-revisited items are censored rather than assigned zero utility. This does not isolate a single item's effect from its companions or selection history.

The additional local test restores the same model, optimizer, and random state, holds companions fixed, and swaps a candidate against a token-length-matched alternative. For 600 candidates, predicted utility has Spearman correlation 0.34 with one-update held-out loss improvement versus 0.19 for raw likelihood. This measures local replacement value, not long-term training causality; analysis-only computation is separate from training overhead.

Q3. Equal-trace isotropic noise and factorization?

Q3(a), noise: Q_iso=tr(Q)/K times the identity is the appropriate total-noise-matched control. Cached-observation replay can diagnose changes in state predictions and sampling rankings, but not the end-to-end training contribution of noise structure. We therefore do not attribute the accuracy gain specifically to correlated noise without a controlled training comparison.

Q3(b), factorization: A fixed 2,000-item replay comparing factorized and joint-covariance filtering gives sampling-score rank correlation 0.97 and top-100 overlap 0.91. This is a conditional sensitivity diagnostic under the same observations and update approximation, not an exact-posterior comparison or a guarantee that training trajectories are unchanged.

Q4. Baseline definitions, simultaneous budgets, and exploration/coverage?

The baseline families are static-difficulty-plus-estimated-utility, ability-relative selection, and selection from cached per-example loss. We acknowledge that the submitted descriptions need full scoring-to-probability equations, unseen/stale-item rules, and actual tuning ranges; these will be provided as executable definitions in the revision.

The intended comparison uses the same optimizer settings and nominal valid-target-token budgets. Exact equality of both realized token counts and optimizer updates cannot be inferred from equal batch sizes or packing-bin size, so we will not claim that from those facts alone. The focused exploration/refresh control gives 61.9/59.9/73.5 for accuracy/AULC/coverage. Coverage is an outcome, not a quantity forced equal by changing the policy.

Q5. Paired gains and evaluation separation?

Accuracy differences are +0.6/+0.7/+0.5 and AULC differences +1.2/+0.9/+1.2 for seeds 0/1/2. Benchmarks are not additional independent training runs. Tuning uses a separate 2,000-item development split and development AULC, with 12 trials per sampler. Final benchmarks are evaluated at checkpoints for learning curves; the final endpoint is reported once per seed. We will replace the ambiguous 'once per seed' wording with this distinction and state that benchmark curves were for reporting, not hyperparameter selection.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion and additional evidence

Thank you to all reviewers for the feedback. The discussion centered on the reliability of the gain over a strong adaptive sampler and whether posterior correction captures useful information about the current learner. We have incorporated the additional analyses into the revised manuscript and summarize the main evidence below.

1. Paired results make the strongest-baseline comparison clearer.

Against Difficulty–utility, MoDiff improves final accuracy by +0.6, +0.7, and +0.5 points across the three seeds. The mean gain is 0.6 points with a sample standard deviation of 0.10; mean AULC improves by 1.1 points. These are modest, consistently positive differences in the observed runs. Three seed pairs still limit statistical inference, as discussed in our follow-up response.

2. Additional controls examine where the benefit comes from.

A Difficulty–utility control matched to MoDiff's exploration floor and refresh interval reaches 61.9% accuracy, compared with 62.4% for MoDiff. This leaves a numerical advantage after matching those sampling settings.

The component ablations provide complementary evidence: removing shared abilities, posterior uncertainty, or dynamics reduces accuracy from 62.4% to 60.4%, 60.7%, and 61.0%, respectively. These results help explain the role of the posterior model beyond the final comparison with one baseline.

3. Independent checks strengthen the interpretation of posterior correction.

On a fixed 1,200-item subset evaluated at frozen checkpoints, correctness AUC is 0.71 for posterior gap/mastery predictions, versus 0.62 for static difficulty and 0.66 for current likelihood. The analysis controls question/skill type, answer format, and length, and the correctness labels remain evaluation-only.

A separate 600-candidate test restores the model and optimizer state and holds companion items fixed. Predicted utility has a Spearman correlation of 0.34 with one-update held-out improvement, compared with 0.19 for raw likelihood. This provides a controlled local check alongside the original observational next-visit analysis.

4. The revision clarifies implementation and evaluation.

The manuscript now lists the 12 skills and reported initialization settings, describes the diagnostic protocols, and separates development-based tuning from benchmark reporting. The contribution remains an adaptive curriculum for spatial SFT with usable skill metadata. It reuses ordinary training likelihoods, adds no VLM parameters or extra VLM forward passes for sampling, and has a measured training overhead of 3.8%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.