- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In multi-modal diffusion models, training needs to choose which tokens of which modality are masked and which are visible. This process implicitly allocates the training budget across different modalities, and usually this is uniform at random. The paper proposes a technique, Multimodal DLM Adaptive Curriculum learning (MaDAC), that explicitly allocates training across the different modalities. The algorithm has an outer and an inner step. The outer step chooses which direction to predict (e.g., from molecule modality to cell image modality) and how much of each modality to mask. It favors choices on which the model's loss on fixed probe data is dropping fastest. A fixed fraction of directions is always kept to prevent forgetting. The inner step re-weights the hidden tokens inside one example, by default giving more weight to tokens the model is already confident about. MaDAC is evaluated on three biology tasks: linking compounds to cell morphology (JUMP), medical image segmentation from a text prompt (BioMed), and tissue segmentation on organ slides (HuBMAP). The authors test their algorithm on two different models: a pretrained LLaDA2.0-Uni (16B) fine-tuned with LoRA, and a 147M model (UniDisc) trained from scratch. MaDAC beats the same model trained with standard random masking in most settings. For example, with LLaDA2, BioMed Dice goes from 0.469 to 0.616 on held-out sources and from 0.775 to 0.860 in-distribution, and HuBMAP held-out Dice goes from 0.387 to 0.431; with UniDisc, JUMP SMILES→morphology Hits@10 goes from 0.486 to 0.564. Gains are smaller elsewhere, and LLaDA2-based JUMP results are mixed.
Critical strengths
I found the idea of treating the multi-modal diffusion training as an allocation problem and building an adaptive curriculum around it interesting and, as far as I know, new.
Some results are strong, especially on BioMed source-held-out segmentation with the pre-trained Llada backbone.
The authors shared their code which was helpful to better understand some aspects of the experimental settings.
Critical weaknesses
I find the overall idea interesting, but a few concerns related to the experiments prevent me from understanding if the algorithm is actually useful in practice. Specifically: the experiments do not seem to clearly test the adaptive part of the algorithm described (and the shared code suggests the reported runs may not use it), and I have concerns related to the fairness of the comparison with baselines. Additionally, from a reproducibly point of view, some information is currently missing.
The paper claims that the adaptive allocation causes significant gains across settings. I find this claim only partially supported by the results shown.
- Without a proper ablation I cannot tell if the gains come from hiding each modality at a separate rate, or from adapting those rates during training (given that MaDAC does both and no baseline or ablation does only one of them). The missing baseline that would answer this is simply to draw each modality's rate from a fixed, separate distribution, with no adaptation. This would be especially relevant on BioMed, where the gains are largest, but it has no ablations at all. Even the ablations on JUMP do not test this, as the only non-adaptive version (“fixed-allocation”) is the uniform baseline itself (identical numbers), which uses one constant setting, equal for both modalities at every step.
- On the HuBMAP task, it seems to me that MaDAC does not use its ability to choose the path, since the task is fixed (?) to predict the mask from the image and the organ name. If that is the case, then the outer step boils down to simply showing harder-samples more often, which is a known technique. If this is correct, then HuBMAP cannot be used to support the paper’s main claim about allocating corruption across modalities.
- In general, results are mix rather than showing consistent and significant gain across settings. I would say that they gains are large and significant for BioMed with Lllada2, small or within standard deviation in many of the other settings, and slightly negative for Llada2 on 2 of 3 JUMP retrieval tasks. So "significant across settings" is only partially supported.
- The code shared raises more concerns about the paper since what seems implemented there is not what is described in the paper. Specifically none of the three tasks seems to support the claim that “MaDAC adapts training distribution in response to the learner state”:
- BioMed's outer level seems to use a pre-set anneal not the algorithm presented in the paper.
- JUMP's seems to use a fixed 50/50 route mix, not the dynamic algorithm presented in the paper.
- HuBMAP has a single route (this was clear also from the paper and confirmed in the code).
I am not sure the set up is completely fair to the baselines
- On HuBMAP, it seems that only MaDAC gets a warm-up phase that is used to train the decoder first. So it is unclear how much of the gain comes from this additional training budget. This additional budget should be given to the baselines too as additional training steps (or as warm-up exactly as for MaDAC).
- MaDAC's controller measures progress on "fixed probe data" but the paper only says it uses “training or calibration data” and not where they are coming from. If it is the calibration/validation set, MaDAC receives a training-time signal from data the baselines don’t receive.
Reproducibility concerns
- Hyperparameter values or tuning budgets are not reported for MaDAC nor for uniform / easy-to-hard. This prevents reproduction from the paper alone and makes it impossible to rule out that gains come from extra tuning.
- Tables report mean and standard deviation, but the paper never says how many seeds were run or what the standard deviation is computed over.
Not reasons for rejection, but worth a discussion.
- Every time I see a curriculum-based work, I am reminded of the paper "When Do Curricula Work?" (2021), which found that curricula help mainly with limited training budgets or noisy data, but almost all benefit disappear with enough compute. Is it the case here too? I guess in order to test this the authors would need: - UniDisc at several sizes trained from scratch, with the same budget; - LLaDA2 with full fine-tuning and longer budgets, to see whether the MaDAC–uniform gap closes; - gap vs. training steps, i.e., Fig. 9-style curves for BioMed and JUMP.
- The format of the paper is very dense: figures and table are packed closely, fonts on figures are too small. All this makes reading the paper a difficult experience.
Questions for authors
- Could you run the ablation described above (fixed, separate per-modality rates with no adaptation) on BioMed and JUMP)?
- Could you explain the discrepancy between the algorithm explained in the paper and the config files that describe the algorithm for the three tasks?
- Can you report results on HuBMAP for a uniform baseline with the same decoder warm-up as MaDAC?
- Can you clarify where the “fixed probe data" used by MaDAC comes from, and confirm that the setting is fair to the baselines?
- Can you list all hyperparameter values and how you found them, for MaDAC and for the baselines?
- Could you clarify how many seeds were run and what is the standard deviation is computed over?
- Can you comment on my concern related to the benefit of curricula disappearing as scale (of data or compute) increases?
13 comments
Response to Reviewer yNzh
Q5: All hyperparameters and how they were found
Tables R6-A–R6-F list the training recipes, MaDAC parameters, baseline switches and selection procedures for JUMP, HuBMAP-MR and BioMed. Shared optimization settings apply to all arms within each task/backbone; Table R6-E specifies the curriculum changes. The implementation records describe the shared training recipes and the common controller constants as fixed in advance. Each table identifies the source of its settings.
Table R6-A — shared MaDAC controller and target-weight parameters
| Parameter | Setting |
|---|---|
| Reference mass ρ | 0.20; q = 0.8π + 0.2p₀. Stationary arms draw every action from p₀ |
| Reference law p₀ | Reference route probabilities π₀; every free corruption rate U(0,1), independent per stream (A2) or one shared draw per route (A1) |
| Route progress | Absolute decrease of the probe loss smoothed by an EMA with coefficient 0.9; standardized across routes; adaptive routing starts after the second probe |
| Route softmax | Temperature τₒ = 1; usage bonus 0.5/(1 + u), u = share of the actions drawn since the previous probe |
| Adaptive rate policy | One Beta per free route/stream (one per route for shared rates); initialized at Beta(1,1); r = 0.2 + 0.8v |
| Rate update | Score-function update, policy learning rate 0.05; reward-baseline EMA 0.9; log shape parameters clipped to [−1.5, 3]; ε = 10⁻⁴ |
| Rate reward | Per-route mean of loss before minus loss after the optimizer step, on the identical corrupted batch with the pre-forward dropout RNG restored; per-record losses average to the training loss |
| HuBMAP / BioMed target weights | w ∝ (1 + γc + κe)·σ((c − b₀)/τw), c = max-class probability, e = 1 − p(gold); γ = 1, κ = 2, b₀ = 0.5, τw = 0.1; detached; normalized to mean 1 over corrupted targets |
| JUMP target weights | w = 1 + wβ·σ((cos − 0.5)/0.15), wβ = 1, on records with SMILES visible and both vectors hidden; normalized to mean 1 |
| HuBMAP example sampling | Strength 1; loss EMA 0.9; multiplier clip [0.5, 2]; ramp starting at 12% of training |
| Replay (B8 / B9) | Every B1 update logs the policy its actions were drawn from. B8 permutes complete probe windows and keeps their internal order; B9 draws each action from the policy of a uniformly chosen B1 update |
Table R6-B — JUMP training recipe, objective and selection
| Setting | UniDisc | LLaDA2 |
|---|---|---|
| Backbone / adaptation | Small: width 768, 12 layers, trained from scratch with ContIO (Tiny 384/6, Base 1024/16, Large 1536/24) | LoRA rank 32, α = 64, dropout 0.05 (attention and shared-expert projections); reserved embedding/head rows and ContIO trainable |
| Optimizer | AdamW, β = (0.9, 0.95); gradient clip 1 | Same |
| Learning rates | Backbone 3×10⁻⁴; ContIO 10⁻³ | LoRA and reserved rows 10⁻⁴; ContIO 10⁻³ |
| Weight decay | Backbone 0.01; ContIO 0 | LoRA 0.01; reserved rows and ContIO 0 |
| Schedule | 200-update warm-up, then cosine on the optimizer-update counter | 150-update warm-up, then cosine |
| Batch | 64 (one GPU) | Global 64 = 4 GPUs × 16 |
| Budget / checkpoints | 36k updates / every 4k | 10k updates / every 2k; longer panel 18k |
| Curriculum objective | Eq. 3 per record (1/max(r, 10⁻³) per hidden stream; vector loss (1 − cos) + 0.5·MSE); InfoNCE weight 0.1, temperature 0.07; bioactivity focal γ = 2 | Same |
| Endpoint terms (every arm, every update) | T7: SMILES → bioactivity, class-balanced CE (positive weight 2), weight 1; T8: SMILES → morphology regression + InfoNCE 0.1/0.07, weight 1 | Same |
| Plain-denoising panel | InfoNCE and focal modulation removed | Same |
| Training probe | 2,048 training compounds drawn with seed 0; refresh every 500 updates | Same |
| Reference corruption | π₀(joint) = 1; independent U(0,1) per-modality rates; whole-modality masking; label hidden with probability 0.5 on direct/joint; guard: ≥ 1 visible input, ≥ 1 hidden target | Same |
| Route probes | direct: SMILES visible, morphology/structure hidden; s2m: structure visible, morphology/SMILES hidden; m2s: morphology visible, structure/SMILES hidden; joint: rate 0.5 per stream | Same |
| Selection | Validation only: composite (mean of T1, T3sm, T2mc Hits@10 and T7 pAUROC) and per-endpoint checkpoints; T8 selected by validation T7 | Same |
Response to Reviewer yNzh
Table R6-C — HuBMAP-MR training and evaluation recipe
| Setting | LLaDA2 | UniDisc |
|---|---|---|
| Backbone / adapter | LoRA on the last 2 layers, rank 8, α = 16, dropout 0.05 | 147M parameters; 12 layers, width 768; trained from scratch |
| Spatial interface | 1,024 image tokens; 16×16 (256) or 32×32 (1,024) mask tokens; organ prompt; residual read-out width 128 | MagViT tokens of 256-pixel tiles; text region 64 + G² tokens holding the organ prompt and G×G mask tokens (G = 16 or 32) |
| Read-out features | Image positions of layers 4/12/20 plus projected mask-position states | Image and mask-query positions of blocks 4/8/12 |
| Optimizer | AdamW, β = (0.9, 0.95), weight decay 10⁻³, gradient clip 1 | AdamW, β = (0.9, 0.95), weight decay 0.05, gradient clip 1 |
| Learning rates / schedule | LoRA 2×10⁻⁵; read-out 3×10⁻⁴; 100-update warm-up, then cosine | Backbone 10⁻⁴; read-out 3×10⁻⁴; 200-update warm-up, then cosine |
| Updates / batch | 12k; batch 2 × accumulation 4 | 12k; batch 8 |
| Mask read-out losses | BCE positive weight 3; Dice 2; prevalence 2; boundary 0.5; corrupted mask cells only | Same |
| Token CE | Eq. 3 with coefficients mask 0.05, organ 0.05, image 0.10 | Same |
| Reference route probabilities | Segmentation 0.40; joint 0.30; image completion 0.15; organ completion 0.15 | Same |
| Canonical probe rates | Segmentation mask 1; organ completion organ 1; image completion image 0.5; joint all streams 0.5 | Same |
| Training probe | 64 iid_train tiles per route / every 200 updates | 256 tiles per route / every 500 updates |
| Selection | Checkpoints every 1k; iid_cal mean native-image Dice; threshold sweep 0.01–0.99 | Same |
| Inference | One forward with the whole mask hidden; image and organ visible; native-resolution reconstruction | Same |
Table R6-D — BioMed recipe
| Setting | Value |
|---|---|
| Backbone / trainable modules | LLaDA2 LoRA rank 32, α = 64, dropout 0.05 on attention qkv/dense, shared-expert gate/up/down, lm_head and word embeddings |
| Optimizer | AdamW, β = (0.9, 0.95), weight decay 0.01 |
| Learning rate / schedule | 2×10⁻⁵; 20-update warm-up, then cosine |
| Batch / hardware | Batch 2 per GPU, accumulation 8; 8× NVIDIA B200 |
| Training horizons | 1× = 30k steps; 2× = 60k steps; checkpoints every 5k |
| Shared initial phase | Segmentation route only for the first 40% of training in every arm, with reference rates |
| Reference patterns | Segmentation 0.25; text reconstruction 0.20; joint mask/text 0.55; image always visible |
| Loss | Eq. 3: 1/max(r, 10⁻³) per stream; two-class mask CE with foreground weight 3; text CE; SoftDice coefficient 5 |
| Training probe | 64 training records per pattern; every 200 optimizer updates; fixed corruption seed 0 |
| Canonical probe rates | Segmentation mask 1; text reconstruction text 0.5; joint mask/text 0.5/0.5 |
| Target weights | Table R6-A; mask cells use the two-class posterior, text tokens the full-vocabulary posterior; applied to mask CE, text CE and Dice |
| Data splits | 16×16 grid, cell foreground at ≥ 25% occupancy; held-out adapter trains on the non-held-out sources' IID training records; held-out sources split into target-calibration (20%) and test records |
| Evaluation | All 256 mask cells hidden; threshold (0.05–0.95) and checkpoint chosen on calibration data: source = iid_cal, target = held-out-source calibration records; held-out adapter: 1,397 held-out and 4,096 IID test records; IID adapter: 11,229 records; global Dice, image Dice, AUROC |
Response to Reviewer yNzh
Dear Reviewer yNzh,
Thank you for your constructive suggestions and insightful comments! We thank the reviewer for the questions about implementation, attribution, fairness and scaling. G.2 defines the common controls; G.5–G.7 report the task results; G.8 explains their implications and data roles. The responses below connect each question to those comparisons. We also provide further clarifications and refinements in the general response and paper revision.
Q1: Fixed, separate per-modality rates without adaptation, on BioMed and JUMP
A2 supplies the requested fixed, separate per-modality rates without adaptation. On BioMed, B1 − A2 improves held-out global Dice by +0.046 / +0.085 at 1× and +0.084 / +0.018 at 2×, in source/target-calibration order. The separately trained IID adapter gains +0.044 [95% CI +0.042, +0.046] (G.7, Tables R1.1–R1.5 and R1-C).
The A1–A4 factorial design answers whether separate rates or their adaptation helps. Adapting separate rates (A4 − A2) gives +0.112 / +0.096 at 1×, +0.078 / +0.048 at 2×, and +0.027 on the separately trained IID adapter. With both policies adapted, separate rates (A4 − A3) give +0.051 / +0.065 on held-out data. Separate fixed rates (A2 − A1) show no consistent held-out gain. On BioMed, separate rates also help (Table R1-R).
JUMP's matched A2 control and full component comparisons appear in G.5 (R2-U/R2-L). B1 − A2 demonstrates all three retrieval endpoints as positive on both backbones. Rate-only adaptation is also compared, showing that its contribution depends on the task and backbone.
Q2: How the submitted configurations relate to the algorithm in the paper
We ran Algorithm 1 on BioMed as B1. It learns route allocation and bounded rate policies using absolute progress and the same-batch reward, with reference mass 0.20 and Eq. 6 target weighting. B1 improves over fixed independent A2 under both calibration rules, at both training budgets, and on the separately trained IID adapter (G.7).
The revised JUMP comparison uses the same B1 definition. HuBMAP-MR supplies four prediction routes under a common read-out/loss interface and additionally uses example sampling.
Q3 and the hard-example concern: matched decoder warm-up, and what HuBMAP measures
HuBMAP-MR gives all matched arms the same training schedule and learning-rate warm-up, with the adapter/backbone trained from the first update. The common interface and schedule are introduced in G.2, and G.6 contains the complete R3 results. The revised comparison therefore matches the training schedule across curriculum arms.
To address the hard-example explanation, B1–B4 retains example sampling and target weighting while fixing allocation: B1 has higher mean image Dice on both backbones. B5/B6 separately remove target weighting/example sampling, and allocation-only B7 falls below A2 in mean image Dice. Thus the evidence identifies what allocation adds to the combined policy and how the gain depends on its supporting factors. G.6 reports these ablations alongside segmentation-only R2/R3.
Mixed results and “significant across settings”
The revised claim specifies the endpoint and comparator. B1 improves BioMed held-out global Dice against A2 under both calibration rules and training budgets, and improves separately trained IID Dice. It also achieves higher mean source-held-out image Dice on HuBMAP-MR and the highest means on all five JUMP endpoints with both backbones. JUMP's reported comparisons resolve all three retrieval endpoints.
The component results explain where the gains arise. BioMed's A1–A4 design identifies rate adaptation as the key contributor, and B1 − B4 improves AUROC under both calibration rules and separately trained IID Dice. HuBMAP-MR's B1 − B4 comparison shows higher mean image Dice with example sampling and target weighting fixed. G.7–G.8 give the endpoint-specific interpretation.
Q4: Where the probe data come from, and fairness to baselines
The training protocol prescribes fixed training-split probes., with exact prescribed settings in R6-A–R6-D below. The comparison uses the stated validation/calibration selection recipe for each arm. Training probes, model selection and the utility audit thus have explicit, separate data roles.