- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MoDiff, an adaptive curriculum for spatial vision-language model fine-tuning. Static difficulty metadata initializes item-difficulty priors, while a dynamic multidimensional item-response model maintains shared spatial abilities and item-specific difficulty factors. Length-normalized target likelihoods from ordinary supervised training provide soft observations for Gaussian filtering. A posterior-averaged local information score then determines sampling probabilities, with an exploration floor. The evaluation uses SpatialLadder-26K, five spatial benchmarks, three backbones, and several training budgets. The reported tight-budget result is 62.4% average accuracy, compared with 59.3% for static scheduling and 61.8% for the strongest matched adaptive baseline, with 3.8% training-time overhead. Ablations and predictive analyses examine the roles of shared abilities, uncertainty, and learner dynamics.
Critical strengths
-
The method provides a coherent and inexpensive way to combine prior difficulty with evolving model behavior. The distinction between a fixed item factor and a shared, changing ability state is useful: observations can revise the estimated difficulty of a visited item while changing predictions for related, unvisited examples. Sections 3.2–3.4 specify the observation mapping, filtering updates, and sampling rule, and Algorithm 1 makes their relationship to ordinary SFT explicit. Reusing target likelihoods without auxiliary VLM evaluations is practically attractive. The paper also appropriately distinguishes its generalized Bayesian observation model and local information surrogate from calibrated semantic correctness and exact posterior information gain.
-
The experiments include relevant controls and informative component ablations. The controlled comparisons match valid target tokens and optimization budgets, distinguish reproduced runs from published SpatialLadder results, and include static, loss-based, ability-adaptive, and difficulty–utility baselines. Table 2 shows appreciable degradation when shared abilities, posterior uncertainty, or dynamics are removed: average accuracy falls from 62.4 to 60.4, 60.7, and 61.0, respectively. Tables A4 and A7 extend the comparison across budgets and model families. These results make the approach promising even though the incremental advantage over the strongest baseline remains uncertain.
Critical weaknesses
W1. The empirical advantage over a strong adaptive sampler is not established robustly, and important comparison details remain unspecified.
- The headline 3.1-point improvement is relative to static scheduling. The more consequential comparison is with Difficulty–utility: the tight-budget difference is 0.6 points in Table 1, and the differences on the other two backbones are 0.5 points in Table A7. Although results average three seeds, no seed-level values, dispersion, or uncertainty on paired differences are reported. The authors explicitly acknowledge this limitation in Section 4.1. The additional AULC improvement is useful, but its variability is also unavailable. Thus, the current evidence does not establish whether the extra posterior machinery delivers a reliable improvement over a strong adaptive alternative.
- Section 4 describes Loss self-paced, Ability-adaptive, and Difficulty–utility only at a high level. Their sampling equations, treatment of unseen items, update/refresh rules, and tuning budgets are not sufficiently specified to assess the strongest comparison. Matching tokens and optimizer settings is necessary, but does not establish that these samplers received comparably effective implementations and tuning.
- Table A2 omits parameters that directly determine MoDiff's inferred uncertainty and sampling behavior, including the item-prior variance, initial ability covariance, and process-noise scales in Equations (2), (3), and (6). The development split, complete skill mapping, and absolute token/update budgets are also unclear. These omissions make the small reported margins difficult to reproduce and interpret. I regard them as potentially fixable, but central to assessing the contribution.
W2. The evidence does not yet validate the central interpretation that posterior correction identifies model-specific spatial difficulty and useful training examples.
- The observation in Equation (4) is a geometric mean of target-token probabilities, including for multiple-choice examples without option normalization. The paper correctly acknowledges that this is not semantic correctness. However, likelihood can also reflect answer wording, tokenization, or response format. The reported posterior reversals therefore do not independently demonstrate that static spatial difficulty has been corrected, as opposed to reorganizing examples according to another likelihood-related property. Sections 4.2 and 4.4 explicitly acknowledge the absence of independent frozen-checkpoint validation.
- Table 3 is the main evidence connecting the inferred state to subsequent outcomes, reporting an AUC of 0.77 for predicting next-visit utility. Yet the early checkpoint, analysis population, held-out loss set, and evaluation procedure are insufficiently described. In particular, “positive held-out loss reduction after the item's next training visit” does not explain how an individual item's contribution is separated from other items in its minibatch, differing visit times, or the selection policy itself. Without these details, the AUC cannot establish that the score predicts an item's training value under comparable conditions.
- Several diagnostics cannot resolve this gap. The reversal subset in Figure 2 lacks its selection thresholds and checkpoint, while Figure 3(c,d) reports window/width sweeps whose settings are explicitly undocumented and do not directly correspond to the implemented refresh interval and temperature. I appreciate the manuscript's qualifications, but those qualifications leave the mechanism supported mainly by internal posterior summaries and component-removal ablations. This limits the additional understanding established by the paper beyond a promising sampling heuristic.
Questions for authors
-
Reliability of the improvement: Could you provide per-seed results and uncertainty on the paired MoDiff versus Difficulty–utility differences, for both final accuracy and AULC? Does the advantage remain consistent across seeds, benchmarks, budgets, and backbones?
-
Adaptive baseline definitions: What are the exact sampling and update rules for Difficulty–utility, Ability-adaptive, and Loss self-paced? How do they handle unseen or stale examples, and what metadata, exploration mechanisms, refresh frequencies, and tuning budgets do they share with MoDiff?
-
Reproduction of the controlled comparison: Please specify the development split, prior and process-noise parameters, complete 12-skill mapping, absolute token/update budgets, and timing hardware. How were these settings selected while keeping benchmark evaluation separate from model and sampler selection?
-
Predictive-validity protocol: How exactly was Table 3 constructed? Please identify the checkpoint, examples, held-out loss set, outcome definitions, and treatment of unvisited items, and explain how next-visit utility is attributed to an individual item when training uses minibatches and adaptive selection.
-
Independent validation of difficulty correction: Can the reported reversal analysis be evaluated using specified, fixed subsets and independent checkpoint measurements of answer correctness? In particular, does the relationship survive controls for answer format or comparisons within the same question/skill type? This would materially strengthen the interpretation of the latent state.
2 comments
Response to nTsY (2/2): questions for authors
Questions For Authors:
Q1. Paired uncertainty and consistency across settings?
The final-accuracy pairs are 62.5/61.9, 62.4/61.7, and 62.3/61.8, giving mean difference +0.6 and sample SD 0.10. AULC differences are +1.2/+0.9/+1.2, with mean +1.1 and sample SD approximately 0.17. Normal-assumption paired t intervals are approximately [0.35,0.85] for accuracy and [0.67,1.53] for AULC. The exact two-sided sign-flip test is 0.25 for either outcome; these methods have different assumptions and should not be conflated.
Existing mean accuracy differences are +0.7 at medium/full budgets and approximately +0.5 on the other two backbones. They are descriptive extensions, not independent seed-level significance evidence. We will report the small CV-Bench margin without asserting a robust benchmark-specific advantage.
Q2. Exact adaptive-baseline rules and shared resources?
Difficulty-utility combines static difficulty with estimated utility; Ability-adaptive uses model-relative ability; Loss self-paced uses cached target losses. We agree that these labels alone are not reproducible definitions. The revision will provide their scoring-to-probability mappings, update schedules, initialization for unseen items, stale-score handling, and development search ranges.
The focused Difficulty-utility control shares MoDiff's exploration floor and refresh interval and scores 61.9 accuracy, 59.9 AULC, and 73.5 coverage. This addresses those two concrete alternatives but does not establish equivalence of every implementation detail. Metadata access and method-specific settings will be disclosed explicitly.
Q3. Development split, parameters, budgets, hardware, and selection separation?
A separate 2,000-item development split is used for development-AULC tuning, with 12 trials per sampler. The tested state uses 12 skills and the initialization listed in W1(c); the full metadata mapping and process coefficients will be included in the configuration specification. Timing uses eight A100-80GB GPUs.
The intended training comparison shares optimizer settings and nominal valid-target-token budgets. Identical packing bins do not alone establish identical realized token and update counts, so we will avoid that inference. Checkpoint benchmark evaluations are used for reporting learning curves; the final endpoint is reported once per seed. They are distinct from the development criterion used for hyperparameter selection. The manuscript's 'once per seed' wording will be corrected to make this explicit.
Q4. Table 3 construction and minibatch attribution?
The original score is recorded at 2,000 updates, with next-visit loss change measured on a fixed 512-item development holdout. The training visit has 15 companion items; never-revisited cases are censored, not assigned zero utility. This outcome is associated with the sampled minibatch and policy history, so its AUC is not isolated item causality.
For the controlled local test, each trial restores model, optimizer, and random state; companions stay fixed; a candidate is swapped against a token-length-matched alternative. Across 600 candidates, correlation with held-out one-update improvement is 0.34 for the utility score versus 0.19 for raw likelihood. This estimates local replacement value. Additional diagnostic computation is separate from training overhead.
Q5. Independent correctness on fixed subsets with format/skill controls?
The 1,200-item subset is fixed independently of reversal status, stratified by skill/format, and evaluated at frozen 25%/50%/100% checkpoints with the released parser. Descriptive AUCs are 0.71/0.62/0.66 for posterior gap/mastery, static score, and likelihood. Comparisons within question/skill and answer-format groups, with length controls, address the proposed confounds. Evaluation labels are never fed into the sampler. This strengthens the limited predictive interpretation without claiming that every posterior reversal reveals a genuine spatial blind spot.
Response to nTsY (1/2): critical weaknesses
Thank you for the careful assessment. We follow W1 and W2, including their constituent concerns.
Critical Weaknesses:
W1. Reliability and controlled-comparison details.
W1(a), paired gains: Seed 0/1/2 final accuracies are 62.5/62.4/62.3 for MoDiff and 61.9/61.7/61.8 for Difficulty-utility. Differences are +0.6/+0.7/+0.5, mean +0.6, sample SD 0.10. A t interval under the normal-difference assumption is [0.35,0.85]; the exact sign-flip p-value is 0.25. Thus, the direction is consistent in these runs, while inference from three pairs remains assumption-sensitive. We will not infer universal superiority or a separate robust CV-Bench gain from the 0.2-point mean gap.
W1(b), baseline fairness: An exploration/refresh-matched Difficulty-utility control reaches 61.9 accuracy, 59.9 AULC, and 73.5 coverage, versus 62.4/60.7/74.0 for MoDiff. Matching these settings leaves +0.5 accuracy and +0.8 AULC points. This addresses the exploration/refresh explanation directly; it does not replace disclosure of the remaining implementation and tuning details. We will include exact sampling equations, unseen/stale policies, and tuning ranges in the revision.
W1(c), reproduction: The state uses 12 metadata-derived skills, normalized multi-tag loadings, prior scale 1.0, clipping parameter epsilon_h=0.05, item variance 0.25, and initial ability covariance 0.5 times the identity. Equation 6 adds isotropic noise and batch-averaged skill outer products. We acknowledge that the full mapping, deterministic prior preprocessing, and complete noise configuration are necessary implementation dependencies. The evidence is conditional on those fixed choices. Budget matching will be described in terms of actual token/update accounting, rather than inferred from identical batch settings.
W2. Independent evidence for difficulty correction and utility.
W2(a), correctness versus likelihood: We added frozen-checkpoint answer-correctness evaluation on 1,200 fixed skill/format-stratified items at 25%/50%/100% of the tight budget. The descriptive AUCs are 0.71 for posterior gap/mastery, 0.62 for the static score, and 0.66 for raw likelihood. Within-group comparisons control question/skill, answer format, and length. The labels do not enter training. This is independent of the soft-likelihood observation, but does not turn that observation into calibrated semantic correctness.
W2(b), Table 3: Ordinary loss change after a minibatch visit is observational, not a single item's causal value. We will label Table 3 accordingly. A reset-state, token-matched replacement test with 600 candidates gives utility-versus-improvement correlation 0.34, compared with 0.19 for raw likelihood. This supports local one-update prediction under fixed companions, not long-term causal utility.
W2(c), exploratory figures: Posterior reversals remain exploratory. We will not use a reversal-selected subgroup as independent validation. Figure 3(c,d) is also not treated as evidence for controlled refresh/temperature effects without a documented correspondence; those panels will be removed from the evidential argument. The new fixed-subset correctness test supplies the independent check instead.