acceptodds
Under review as a conference paper at ICLR 2027

CoRAL Synthesis: Learner-Relative Calibration of Data Complexity Composition for Improving VLM Generalization

Abstract

The data complexity of training examples can substantially affect optimization and generalization. Yet, data complexity remains an underspecified axis of synthetic data design, with complexity typically controlled using simple heuristics rather than principled criteria. This raises two questions: how data complexity affects downstream performance and how it should be leveraged to promote generalization. To study this in vision-language tasks, we construct hundreds of controlled VQA post-training settings spanning multiple complexity levels, using matched distinct subsets that isolate complexity from data scale, quality, and diversity. We find that complexity substantially affects out-of-distribution (OOD) generalization under a controlled setting, with some smaller subsets outperforming larger training sets. Moreover, our empirical analysis shows that appropriately mixing Easy, Medium, and Hard examples improves generalization over relying on any single complexity level, but realizing this benefit requires defining these levels relative to the target learner. We also find that existing complexity measures provide limited guidance for synthetic data curation. Building on these findings, we introduce CoRAL Synthesis, a scalable framework that operationalizes learner-relative complexity to govern synthetic data composition. CoRAL Synthesis consistently outperforms naive few-shot synthesis across data scales, with its 1K setting surpassing the vanilla 16K setting. Using CoRAL Synthesis, we release CoRAL-GeneralVQA and CoRAL-ReasoningVQA, two large-scale synthetic datasets for vision-language reasoning. Models trained on these datasets achieve strong OOD generalization; in particular, CoRAL-ReasoningVQA-4B outperforms the strongest evaluated 4B baseline by 5.30%p on average across 9 unseen vision-language reasoning benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Data Complexity, Synthetic Data, Data Curation, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZjBzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the impact of the complexity of synthetic VQA data on the generalization of VLMs during post-training. The authors use the number of correct rollouts from the target learner as a learner-relative complexity signal. By training models while controlling for dataset size, quality, and diversity, this paper draws 3 main conclusions: (i) complexity affects OOD performance; (ii) mixing Easy, Medium, and Hard samples outperforms using a single difficulty level, and complexity is learner-relative; and (iii) existing complexity metrics have limited predictive power. Building on these findings, they propose CoRAL Synthesis, which combines gradient-space diversification and double-critic quality filtering. They release two datasets and report OOD performance gains and scalability results on small models.

Critical strengths
  1. The research question is valuable. Complexity is an overlooked factor in synthetic data design. The problem is clearly defined. The study aims to examine complexity separately from data scale, quality, and diversity.

  2. The paper conducts extensive experiments with broad coverage. These efforts are commendable.

  3. The released datasets could benefit the research community.

Critical weaknesses
  1. This paper misses key ablation in the final system and the gains cannot be attributed to complexity calibration. The paper's main claim is that learner-relative complexity calibration improves performance. However, the scaling comparison in section 4.2 evaluates the full CoRAL pipeline against vanilla few-shot generation, and the main results in Tab. 3 compare against other open VLMs. Neither of them isolates calibration. It is hard to attribute the performance gains

  2. In the controlled experiment for the learner-relative claim, the pattern disappears when labels from Qwen3.5-0.8B are transferred to LFM2.5-VL-450M. However, LFM is a weaker model with a different architecture. Its performance noise may overwhelm CMG differences.

  3. The method's novelty appears limited. Several components, such as gradient-space diversification, two-stage generation, and Double-Critic filtering are all prior work. It is also mentioned by authors.

  4. The scalability claim does not account for the computational cost. Each candidate requires rollouts from the target learner, LLM-based semantic verification calls, and critic calls. The paper provides no computational cost comparison with the baseline. It also does not evaluate the low-cost alternatives proposed in the Discussion.

Questions for authors

See weakness

6 comments
AuthorsOpenReview ↗

Responses to Reviewer ZjBz (1/2)

We sincerely thank the reviewer for the positive evaluation of our work and the thoughtful feedback! We appreciate their recognition of complexity as an overlooked yet important factor in synthetic data design, our extensive controlled experiments, and the potential value of our released datasets to the research community.

The reviewer’s critiques have helped us clarify our methodological contributions and further strengthen the empirical evidence supporting our work.

Below, we respond to each concern with targeted analyses, supplementary experiments, and clarifications.


W1: Contribution of Individual CoRAL Components

We are grateful to the reviewer for this insightful suggestion, which motivated us to conduct additional component-wise ablations to disentangle the contributions of each components.

Beyond the requested ablation, we further evaluate all possible combinations of

  • Quality (Q, Double-Critic filtering),
  • Diversity (D, gradient diversification), and
  • Complexity (C, complexity calibration).

Following the original vanilla synthesis comparison, we control for data scale and component combinations, and report the performance gain (Δ) from adding complexity calibration (C), as shown below.

Component1K / Δ(C)4K / Δ(C)8K / Δ(C)16K / Δ(C)64K / Δ(C)
Vanilla64.2364.6864.0265.1265.00
Q64.6964.2064.6064.8864.94
D64.7564.8565.2565.6666.00
C63.97 / -0.2663.85 / -0.8364.31 / +0.2965.55 / +0.4365.02 / +0.02
Q+C64.71 / +0.0264.86 / +0.6664.98 / +0.3865.68 / +0.8065.59 / +0.65
D+C64.89 / +0.1465.47 / +0.6265.85 / +0.6066.05 / +0.3966.22 / +0.22
Q+D64.4665.2265.7865.9766.06
Q+D+C (= CoRAL)65.28 / +0.8265.51 / +0.2965.91 / +0.1366.18 / +0.2166.99 / +0.93

For a clearer comparison, we encourage the reviewer to refer to Fig. 10 in the revision.

Our ablation results reveal the following findings.

  • Complexity calibration (C) complements quality (Q) and diversity (D). Adding C consistently improves performance when combined with Q, D, or Q+D across all five scales, highlighting the complementary benefits of jointly considering these factors.
  • However, C alone shows inconsistent improvements, aligning with prior findings [1] that quality control can be important for effectively leveraging complexity calibration.

Overall, these results confirm the consistent incremental contribution of complexity calibration beyond quality and diversity.

We have incorporated these component-wise ablation results to §B.4 (lines 1986–1992) and included a reference in the main text (lines 477–479).

Reference:

[1] Davidson et al., 2026. Reasoning-Driven Synthetic Data Generation and Evaluation.


W2: Reliability of Learner-Relative Analysis

We thank the reviewer for this valuable comment. Our original analysis intentionally examined learner-relative effects across models with different capacities and architectures to better reveal the importance of learner-specific complexity calibration. We also recognize the value of comparing self-assigned and transferred labels under a fixed target model to further validate these findings. We therefore conduct an additional controlled experiment.

We compare single-complexity-level performance and CMG across five representative compositions, using InternVL3.5-1B, for which a learner-specific correctness datastore is already available, given time constraints during the rebuttal period.

Below is the results. Each cell reports Self-label (i.e., learner-specific) → Transferred-label (labels transferred from Qwen3.5-0.8B).

OOD Performance in General-VQA.

CompositionEpoch 1Epoch 2Epoch 3Epoch 4Epoch 5
E458.41 → 59.35 (+0.94)59.13 → 59.45 (+0.32)58.83 → 59.56 (+0.73)58.83 → 59.30 (+0.47)58.93 → 59.73 (+0.80)
M459.50 → 59.66 (+0.16)58.45 → 59.46 (+1.01)58.46 → 58.92 (+0.46)58.65 → 59.18 (+0.53)58.62 → 59.23 (+0.61)
H458.52 → 57.61 (−0.91)58.05 → 57.69 (−0.36)58.35 → 57.77 (−0.58)58.29 → 57.94 (−0.35)58.34 → 57.87 (−0.47)
E1H358.51 → 59.21 (+0.70)58.88 → 58.37 (−0.51)58.80 → 58.20 (−0.60)58.31 → 57.49 (−0.82)58.55 → 58.44 (−0.11)
E2M1H158.86 → 57.61 (−1.25)59.03 → 57.73 (−1.30)58.53 → 59.05 (+0.52)59.01 → 58.05 (−0.96)58.77 → 58.13 (−0.64)

W2 follow in our second comment.

AuthorsOpenReview ↗

Responses to Reviewer ZjBz (2/2)

W2: Reliability of Learner-Relative Analysis (Continued)

Complexity Mixture Gain (CMG)

CompositionEpoch 1Epoch 2Epoch 3Epoch 4Epoch 5Mean CMG
E1H3+0.018 → +1.165 (+1.148)+0.560 → +0.240 (−0.320)+0.330 → −0.018 (−0.348)−0.115 → −0.790 (−0.675)+0.063 → +0.105 (+0.043)+0.171 → +0.141 (−0.030)
E2M1H1+0.150 → −1.383 (−1.533)+0.340 → −1.283 (−1.623)−0.088 → +0.098 (+0.185)+0.360 → −0.880 (−1.240)+0.065 → −1.010 (−1.075)+0.166 → −0.892 (−1.058)

The results reveal two observations:

  1. Single-level vs. mixed composition. With transferred labels, mixed compositions fail to outperform the best single-level baseline across all five epochs. The previously observed benefits of E1H3 over H4 also become less consistent.
  2. Composition gains across training. Transferring labels substantially changes the CMG trends across training dynamics. For E2M1H1, positive CMG decreases from 4/5 to 1/5 epochs, with mean CMG dropping from +0.166 to −0.892.

These findings further support our original conclusion that effective complexity composition should be calibrated relative to the target learner.


W3: Novelty of Method

We thank the reviewer for raising this concern. We agree that several components build on prior work, as cited in our paper. However, we would like to clarify the distinction between the tools we employ and our main contribution.

  • When introducing a synthesis framework, accounting for these well-established data factors is essential. We use established components as tools rather than claiming novelty in them. We intentionally employ well-established techniques to reliably operationalize quality (Q) and diversity (D), rather than claiming novelty in these individual components.
  • Beyond Q+D to Q+D+C in end-to-end synthesis. Our method’s main contribution is explicitly incorporating learner-relative complexity (C) alongside Q and D into an end-to-end synthesis framework, extending conventional Q+D-oriented synthesis.

We believe this represents a meaningful contribution to synthetic data generation beyond simply combining existing components.


W4: Computational Cost Analysis

We thank the reviewer for this insightful comment. We agree that the additional computational cost of complexity calibration should be discussed.

To provide a reliable comparison, we empirically measure the GPU-hours required for each component of Vanilla 16K, CoRAL 1K, and CoRAL 1K w/o complexity calibration. We account for seven computational stages:

  • Data Generation: Problem Generation (27B), Solution Generation (27B)
  • Diversification & Quality Control: Gradient Diversification (0.8B), Double-Critic (27B)
  • Complexity Calibration: Learner Rollouts (0.8B, N=8), Semantic Verification (27B)
  • Training: Fine-tuning (2 epochs)

Following our experimental setup, we use H200 GPUs for 27B components and RTX 6000 Ada GPUs for the smaller-model operations and fine-tuning. The resulting GPU-hours and estimated costs are summarized below.

ComponentVanilla 16KCoRAL 1KCoRAL 1K w/o ComplexityGPU
Problem Generation (27B)0.2660.1290.035H200
Solution Generation (27B)3.9931.9390.525H200
Gradient Diversification (0.8B)—1.0790.292RTX 6000 Ada
Double-Critic (27B)—0.2880.078H200
Learner Rollouts (0.8B, N=8)—1.881—RTX 6000 Ada
Semantic Verification (27B)—0.367—H200
Model Fine-tuning (2 epochs)2.0540.1530.153RTX 6000 Ada
Total H200 Hours4.2592.7230.638
Total RTX 6000 Ada Hours2.0543.1130.445
Estimated Cost (USD)$16.81$12.08$2.62

Cost estimates use ‘RunPod Community Cloud’ pricing (November 2026): H200 at USD 3.59/hour and RTX 6000 Ada at USD 0.74/hour.

The results show that scaling up vanilla synthesis incurs higher costs primarily from 27B solution generation and model fine-tuning due to data scale. Although complexity calibration incurs additional computational costs through iterative processing, we consider this inference-time overhead a reasonable trade-off for generating high-quality synthetic data, consistent with prior studies that invest additional computation in data synthesis [1, 2].

—

The low-cost alternative in §Discussion was intended as a potential future direction rather than a validated method. We believe that a direct empirical comparison of computational efficiency provides more meaningful insights, and have therefore revised the §Discussion (lines 843-866) accordingly. We hope this clarification addresses the reviewer's concern.

Reference:

[1] Davidson et al., 2026. Reasoning-Driven Synthetic Data Generation and Evaluation.

[2] Jung et al., 2026. DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation.

AuthorsOpenReview ↗

Follow up with reviewer ZjBz

Dear Reviewer ZjBz,

We sincerely appreciate your thoughtful and constructive feedback. We provided a detailed response addressing each of your concerns, including comprehensive component-wise ablations, additional controlled experiments on learner-relative complexity calibration, clarifications of our methodological contributions, and a detailed computational cost analysis.

Given the approaching deadline, we remain fully committed to addressing any remaining questions you may have and would be grateful for any further feedback on these additional analyses.

Thank you again for your time and valuable suggestions, which have significantly strengthened our work.

Reviewer_5QutOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper claims that data complexity is a critical dimension in synthetic data curation for Vision-Language Models (VLMs). The authors show that mixing Easy, Medium, and Hard examples in a suitable ratio yields superior out-of-distribution (OOD) generalization compared with training on any single complexity stratum. Leveraging these insights, the authors introduce CoRAL Synthesis, an end-to-end synthetic data generation framework. Using CoRAL Synthesis, they generate two million-scale datasets: CoRAL-GeneralVQA and CoRAL-ReasoningVQA. On 4B-parameter open VLMs, models trained on CoRAL data demonstrate substantial OOD generalization gains.

Critical strengths
  1. The paper demonstrates that static complexity metrics fail to predict downstream generalization, which provides a valuable insight for the data-centric AI literature.

  2. The authors translate their empirical findings into a data synthesis framework that demonstrates impressive sample efficiency. The released million-scale datasets (CoRAL-GeneralVQA and CoRAL-ReasoningVQA) yield state-of-the-art results among 4B-scale VLMs in terms of OOD generalization.

Critical weaknesses
  1. The core idea of this paper (improving OOD generalization by filtering training data by difficulty) and its data-difficulty stratification scheme are highly similar to those of an existing paper [1], and the paper lacks discussion of this related work. After comparing the two papers, I find that the main difference is merely a different mixing ratio for data of different difficulty levels. I do not consider this a sufficiently valuable contribution because this ratio scheme lacks generality (see Weakness 2).

  2. While the paper convincingly demonstrates that mixing complexity levels is beneficial, the optimal Easy/Medium/Hard (E/M/H) ratios vary across domains (e.g., E1/M0/H3 for General VQA vs. E3/M1/H0 for Reasoning VQA). These ratios are currently determined through empirical grid search, and there is no analytical framework for predicting optimal compositions.

[1] Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Training. CVPR2026

Questions for authors
  1. Could the authors provide a detailed comparison with the paper [1] I mentioned and clarify the unique contributions of this paper?

  2. Did the authors explore dynamic or online complexity adjustment during training rather than maintaining a static pre-selected mixture ratio?

4 comments
AuthorsOpenReview ↗

Responses to Reviewer 5Qut (1/2)

We sincerely appreciate the time and effort you invested in reviewing our work. Your comments have helped us better clarify our contributions relative to prior work, more precisely define the scope of our findings, and explore a new methodological direction for complexity adjustment.

We address each of these concerns individually below.


W1 & Q1: Contribution of This Research

We thank the reviewer for bringing DC-SFT [1] to our attention. We will explicitly cite and discuss this work in the revised manuscript (line 135-136 in §2) and clarify our novelty relative to it. We respectfully clarify two fundamental differences between the studies.

  1. Our work addresses a fundamentally different research problem from [1], moving beyond difficulty-based data selection to complexity-aware synthetic data generation.
  2. The main distinction is not a specific mixing ratio, but in the broader research objectives and contributions.

—

The research questions, approaches, and contributions of the two studies differ substantially.

  • Research Question

    • [1]: Does RL's superior OOD generalization arise from its implicit focus on medium-difficulty training samples?
    • Ours: How does data complexity affect downstream generalization, and how can it be leveraged in synthetic data generation?
      • As our work is fundamentally about data synthesis, we are confronted with additional challenges unlike [1]:

        1. How can we isolate data complexity effects in synthetic vision-language data while controlling scale, quality, and diversity?
        2. How can we jointly account for these factors in synthetic data generation while calibrating complexity relative to the target learner?

        Our work primarily addresses these challenges, which are beyond the scope of [1].

  • Approach

    • [1]: Curate off-the-shelf datasets through rollout-based difficulty filtering for SFT.
    • Ours: Generate new problem–solution pairs with gradient diversification, quality verification, and learner-relative complexity calibration.
  • Contribution

    • [1]: Explain the RL–SFT generalization gap and propose DC-SFT method, outperforming RL in OOD generalization.
    • Ours: Establish complexity as an independent axis of synthetic data design and introduce CoRAL Synthesis with two large-scale VQA datasets.

—

The closest overlap with [1] lies in our §3.3, where both studies use rollout-based difficulty estimation. However, even within this analysis, the two studies differ in task coverage, composition strategies, and key findings.

We further explain the generality of composition ratios in W2.

—

Taken together, these differences highlight that the two studies address distinct research problems.

Reference:

[1] Lu et al., 2026. Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Training.


W2: Generalizability of Complexity Composition Ratio

We appreciate the reviewer raising this insightful comment. We respectfully argue that identifying a universally optimal complexity composition is inherently challenging [1], as it depends on multiple interacting factors, including the target learner, task characteristics, and training dynamics (§3.3). Precisely because of this challenges, we empirically examine diverse E/M/H 16 compositions under controlled settings, which we believe is a meaningful contribution of our work.

Importantly, recognizing this concern, we deliberately avoid adopting these task-specific optima in our final synthesis pipeline.

  • Instead, we select E2/M1/H1, which achieves positive CMG in 9 of 10 task–epoch settings, prioritizing consistency across tasks and training stages rather than maximizing performance in a particular setting.
  • We apply this shared composition to our synthetic datasets spanning 4 VQA domains (perception & math, science, and chart reasoning).
  • The resulting models demonstrate strong OOD generalization across diverse benchmarks (Tab. 3), while the pipeline exhibits consistent scaling gains over vanilla synthesis (Fig. 1).

These results support the generalizability and practical utility of a shared, empirically motivated composition without assuming a universally optimal ratio.

Reference:

[1] Kordi et al., 2026. Revisiting Generalization Across Difficulty Levels: It’s Not So Easy.

AuthorsOpenReview ↗

Responses to Reviewer 5Qut (2/2)

Q2: Dynamic Complexity Adjustment

We thank the reviewer for this valuable question. While we recognize dynamic complexity adjustment as a promising direction, we would like to clarify that our primary focus is complexity-aware synthetic data design, rather than training-time optimization. Nevertheless, recognizing the value of this direction, we conducted an additional analysis to explore training-time adaptation.

—

1. Exploring dynamic selection objectives. Identifying effective compositions during training may involve different selection objectives, such as loss or gradient-based criteria. We adopt a validation-loss-guided strategy to dynamically select E/M/H compositions during training.

—

2. Experimental setup. We compare the original static E2/M1/H1 composition with the dynamic strategy under the following controlled settings.

  • The same 1M synthetic candidate pool, fixing the training scale (=64K) per epoch and controlling quality and diversity as in our original experiments.
  • We construct a held-out validation set of ~3K samples uniformly drawn from the correctness datastore, separately from the training data.
  • Every 1,000 training steps (1 epoch = 2,000 steps), we evaluate 16 E/M/H compositions on a separate validation set, select the composition with the lowest validation loss, and use it to construct the training data for the next 1,000 steps.

—

3. Results and observations.

The resulting OOD performance is summarized below.

EpochStatic E2/M1/H1DynamicΔ (Dynamic − Static)
165.1365.37+0.24
266.3365.81−0.52
366.0166.31+0.30
465.6565.99+0.34
566.6165.73-0.88

The dynamic strategy selects different compositions as training progresses.

Training Interval (Epoch)Selected CompositionValidation Loss
0–0.5M40.8149
0.5–1E1/M1/H20.7298
1–1.5M3/H10.7076
1.5–2E3/M10.6974
2–2.5M3/H10.6931
2.5–3E2/H20.6910
3-3.5E1/M1/H10.6909
3.5-4E2/M1/H10.6900
4-4.5E3/M10.6895
4.5-5E40.6892
  • Selected complexity ratios. The model tends to favor Medium-heavy compositions early in training and Easy-heavy compositions later, while Hard examples show no clear trend.
  • Static vs. dynamic composition. Neither strategy consistently outperforms the other in OOD generalization, making it difficult to conclude that one is inherently superior.
  • Dynamic adjustment as a complementary direction. Its effectiveness may depend on the selection objective (e.g., loss or gradients) and optimization strategy, warranting further methodological investigation beyond our focus on complexity-aware synthetic data design.

Our exploration suggests that dynamic complexity adjustment is a promising direction, but its effectiveness depends on the selection objective, warranting further methodological investigation.

I thank the authors for their response. However, it does not fully resolve my novelty concern. In particular, DC-SFT also studies how data complexity affects downstream generalization and its potential application to synthetic data generation, and the contribution of this paper is largely incremental.

Given that the overall quality of the paper is reasonable, a score of 1 may be overly harsh. Nevertheless, in light of the limited novelty, I cannot recommend acceptance. I am therefore raising my score to weak reject. I would also encourage the other reviewers and the AC to compare this paper with the related work I mentioned so that its novelty can be assessed more reliably.

AuthorsOpenReview ↗

Responses to Reviewer 5Qut

We sincerely thank the reviewer for reconsidering the assessment and raising the score. We would like to further clarify how our work differs from [1] from the perspective of synthetic data design.

  1. [1] does not consider problem generation, which is a fundamental aspect of synthetic data research.
    • Generating new problems from scratch requires more careful design beyond simple data curation, as it introduces additional challenges in controlling data quality, diversity, and complexity.
      • This is supported by the component ablation results discussed in Point 2.
    • [1]’s synthesis experiment generates reasoning responses to existing questions (a.k.a. distillation).
    • Our work additionally generates new image-conditioned problems, making the problem distribution itself part of data design. For example, we address the following research questions that [1] does not resolve:
      • How much supervision should be generated per image?
        • We empirically identify domain-specific saturation problem in gradient diversity and determine appropriate generation budgets (8/12/8/4 for General/Math/Science/Chart VQA).
        • This is one of the concrete empirical contribution that can inform future synthetic data research.
  2. We agree that data curation methods, including difficulty-based filtering, have potential applications to synthetic data generation, but this alone does not establish how to construct effective synthetic data, which remains a non-trivial challenge.
    • Our component-wise ablations (Fig. 10, lines 1986-1992) illustrate why synthetic data design requires careful consideration.
      • If complexity-based curation alone ([1]) were sufficient in synthetic data generation, it should consistently improve upon naive generation as the dataset scales.
      • Even the C-only vs. vanilla ablation shows that difficulty-based filtering alone is insufficient to achieve consistent scaling gains, a key requirement for an effective synthesis pipeline.
    • These findings provide empirical evidence beyond [1], highlighting the distinct research challenges and value of studying complexity-aware synthetic data design.

Therefore, synthetic data design warrants independent investigation, and we believe our CoRAL Synthesis makes a meaningful contribution.

Reference:

[1] Lu et al., 2026. Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Training.

Reviewer_LcFBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how data complexity affects out-of-distribution (OOD) generalization in vision-language model (VLM) post-training. By evaluating over 300 models across controlled subsets (isolating complexity from scale, quality, and diversity), the authors show that mixing Easy, Medium, and Hard (E/M/H) examples outperforms relying on any single difficulty tier, provided complexity is calibrated relative to the target learner rather than via static proxies. Building on these insights, they introduce CoRAL Synthesis, an iterative framework combining gradient-space diversification, Double-Critic quality verification, and learner-relative complexity calibration (N=8N=8 rollouts, targeting a 2:1:1 E/M/H ratio). They also release two large-scale datasets (CoRAL-GeneralVQA at 624K and CoRAL-ReasoningVQA at 569K), demonstrating that their 1K curated set beats 16K vanilla synthesis and that their 4B reasoning model improves average OOD accuracy by 5.30%p over strong baselines.

Critical strengths
  1. The study cleanly isolates complexity by holding scale constant, applying uniform LM-judge filtering, and matching text, image, and multimodal diversity via Vendi scores across over 300 trained subsets.

  2. Achieves 16x data efficiency over vanilla few-shot generation (1K CoRAL beats 16K vanilla) and yields a +5.30%p average boost on 9 unseen reasoning benchmarks at the 4B scale.

Critical weaknesses
  1. CoRAL Synthesis combines gradient-space diversification, Double-Critic filtering, and 2:1:1 complexity calibration. In Fig. 1 (Right) and Tab. 3, it is unclear how much of the gain over vanilla synthesis comes from complexity calibration versus Double-Critic filtering or gradient clustering.

  2. Estimating complexity requires N=8N=8 rollouts plus a 27B semantic verifier pass per candidate. Furthermore, while complexity is calibrated to the target learner (e.g., Qwen3-VL-4B-Instruct), gradient diversification relies on a smaller proxy (Qwen3.5-0.8B), leaving cross-scale gradient alignment unexamined.

Questions for authors
  1. In the scaling comparison (Fig. 1 Right) and 4B main results (Tab. 3), can you provide an ablation isolating the exact contribution of learner-relative complexity calibration when added on top of gradient diversification + Double-Critic filtering?

  2. How does the total compute cost (synthesis + N=8N=8 rollouts + 27B semantic verification) of curating the 1K CoRAL dataset compare to simply generating and training on the 16K vanilla dataset?

4 comments
AuthorsOpenReview ↗

Responses to Reviewer LcFB (1/2)

We sincerely appreciate the reviewer’s time and effort, and particularly their recognition of our rigorous controlled analysis of data complexity and the effectiveness of CoRAL Synthesis.

We would also like to highlight that the reviewer’s concerns have helped us clarify key aspects of our work and significantly strengthen the paper.

Below, we address each of your questions:


W1 & Q1: Contribution of Individual CoRAL Components

We are grateful to the reviewer for this insightful suggestion, which motivated us to conduct additional component-wise ablations to disentangle the contributions of each components.

Beyond the requested ablation, we further evaluate all possible combinations of

  • Quality (Q, Double-Critic filtering),
  • Diversity (D, gradient diversification), and
  • Complexity (C, complexity calibration).

Following the original vanilla synthesis comparison, we control for data scale and component combinations, and report the performance gain (Δ) from adding complexity calibration (C), as shown below.

Component1K / Δ(C)4K / Δ(C)8K / Δ(C)16K / Δ(C)64K / Δ(C)
Vanilla64.2364.6864.0265.1265.00
Q64.6964.2064.6064.8864.94
D64.7564.8565.2565.6666.00
C63.97 / -0.2663.85 / -0.8364.31 / +0.2965.55 / +0.4365.02 / +0.02
Q+C64.71 / +0.0264.86 / +0.6664.98 / +0.3865.68 / +0.8065.59 / +0.65
D+C64.89 / +0.1465.47 / +0.6265.85 / +0.6066.05 / +0.3966.22 / +0.22
Q+D64.4665.2265.7865.9766.06
Q+D+C (= CoRAL)65.28 / +0.8265.51 / +0.2965.91 / +0.1366.18 / +0.2166.99 / +0.93

For a clearer comparison, we encourage the reviewer to refer to Fig. 10 in the revision.

Our ablation results reveal the following findings.

  • Complexity calibration (C) complements quality (Q) and diversity (D). Adding C consistently improves performance when combined with Q, D, or Q+D across all five scales, highlighting the complementary benefits of jointly considering these factors.
  • However, C alone shows inconsistent improvements, aligning with prior findings [1] that quality control can be important for effectively leveraging complexity calibration.

Overall, these results confirm the consistent incremental contribution of complexity calibration beyond quality and diversity.

We have incorporated these component-wise ablation results to §B.4 (lines 1986–1992) and included a reference in the main text (lines 477–479).

Reference:

[1] Davidson et al., 2026. Reasoning-Driven Synthetic Data Generation and Evaluation.

AuthorsOpenReview ↗

Responses to Reviewer LcFB (2/2)

W2 & Q2: Computational Efficiency and Proxy Reliability

We sincerely thank the reviewer for this insightful comment. While we agree that the additional computational cost of complexity calibration should be discussed against scaling up vanilla synthesis, we would also like to clarify our rationale for using a smaller proxy model for gradient-based diversification.

To provide a reliable comparison, we empirically measure the GPU-hours required for each component of Vanilla 16K, CoRAL 1K, and CoRAL 1K w/o complexity calibration. We account for seven computational stages:

  • Data Generation: Problem Generation (27B), Solution Generation (27B)
  • Diversification & Quality Control: Gradient Diversification (0.8B), Double-Critic (27B)
  • Complexity Calibration: Learner Rollouts (0.8B, N=8), Semantic Verification (27B)
  • Model Training: Fine-tuning (2 epochs)

Following our experimental setup, we use H200 GPUs for 27B components and RTX 6000 Ada GPUs for the smaller-model operations and fine-tuning. The resulting GPU-hours and estimated costs are summarized below.

Component (GPU-hours)Vanilla 16KCoRAL 1KCoRAL 1K w/o ComplexityGPU
Problem Generation (27B)0.2660.1290.035H200
Solution Generation (27B)3.9931.9390.525H200
Gradient Diversification (0.8B)—1.0790.292RTX 6000 Ada
Double-Critic (27B)—0.2880.078H200
Learner Rollouts (0.8B, N=8)—1.881—RTX 6000 Ada
Semantic Verification (27B)—0.367—H200
Model Fine-tuning (2 epochs)2.0540.1530.153RTX 6000 Ada
Total H200 Hours4.2592.7230.638
Total RTX 6000 Ada Hours2.0543.1130.445
Estimated Cost (USD)$16.81$12.08$2.62

Cost estimates use ‘RunPod Community Cloud’ pricing (November 2026): H200 at USD 3.59/GPU-hour and RTX 6000 Ada at USD 0.74/GPU-hour.

The results show that scaling up vanilla synthesis incurs higher costs primarily from 27B solution generation and model fine-tuning due to data scale. Although complexity calibration incurs additional computational costs through iterative processing, we consider this inference-time overhead a reasonable trade-off for generating high-quality synthetic data, consistent with prior studies that invest additional computation in data synthesis [1, 2].

We have added this computational cost analysis to the revised manuscript in the §Discussion (lines 843–866).

—

Our gradient diversification follows Prismatic Synthesis [3], which provides empirical evidence supporting the use of smaller proxy models for gradient-based diversity estimation:

  • §2.3.3 (Cross-Proxy Consistency): Gradient-diversity (=G-Vendi) estimates exhibit strong rank correlations across different proxy models (Spearman's ρ = 0.811–0.899), suggesting that gradient diversity estimation is not highly sensitive to the choice of proxy model.
  • §2.3.4 (Cross-Scale Generalization): Gradient-diversity estimated using a 0.5B proxy remains informative for OOD generalization across student models up to 8B, indicataing that the proxy and target student need not be the same model.

We further compare cross-scale alignment by using both Qwen3.5-0.8B and Qwen3-VL-4B-Instruct as proxy models for gradient-based diversification:

ProxyTrain Model1K4K8K16K64K
Qwen3.5-0.8BQwen3-VL-4B-Instruct71.6871.7871.8372.1472.51
Qwen3-VL-4B-InstructQwen3-VL-4B-Instruct72.0072.0572.1372.2272.81

Using the target model itself yields slightly higher performance, but the smaller proxy still preserves consistent scaling gains. This supports our use of a lightweight proxy for gradient-based diversification in VLM setting, in line with the findings of [3].

This cross-scale proxy comparison is provided in §B.2 of the revision (lines 1820–1824).

Reference:

[1] Davidson et al., 2026. Reasoning-Driven Synthetic Data Generation and Evaluation.

[2] Jung et al., 2026. DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation.

[3] Jung et al., 2025. Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning.

AuthorsOpenReview ↗

Follow up with reviewer LcFB

Dear Reviewer LcFB,

We are grateful for your positive evaluation and constructive feedback. We have submitted a comprehensive response addressing your concerns, including component-wise ablations, a detailed computational cost comparison, and additional cross-scale proxy experiments. These results have also been incorporated into the revised manuscript.

Given the approaching deadline, we would appreciate any further feedback or remaining questions you may have.

Thank you again for your time and constructive feedback!

AuthorsOpenReview ↗

Follow up with reviewer LcFB

Dear Reviewer LcFB,

As the discussion period is closing shortly, we wanted to gently follow up to see if our responses and revisions have addressed your concerns. A brief confirmation would be greatly appreciated.

Thank you again for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.