- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the impact of the complexity of synthetic VQA data on the generalization of VLMs during post-training. The authors use the number of correct rollouts from the target learner as a learner-relative complexity signal. By training models while controlling for dataset size, quality, and diversity, this paper draws 3 main conclusions: (i) complexity affects OOD performance; (ii) mixing Easy, Medium, and Hard samples outperforms using a single difficulty level, and complexity is learner-relative; and (iii) existing complexity metrics have limited predictive power. Building on these findings, they propose CoRAL Synthesis, which combines gradient-space diversification and double-critic quality filtering. They release two datasets and report OOD performance gains and scalability results on small models.
Critical strengths
-
The research question is valuable. Complexity is an overlooked factor in synthetic data design. The problem is clearly defined. The study aims to examine complexity separately from data scale, quality, and diversity.
-
The paper conducts extensive experiments with broad coverage. These efforts are commendable.
-
The released datasets could benefit the research community.
Critical weaknesses
-
This paper misses key ablation in the final system and the gains cannot be attributed to complexity calibration. The paper's main claim is that learner-relative complexity calibration improves performance. However, the scaling comparison in section 4.2 evaluates the full CoRAL pipeline against vanilla few-shot generation, and the main results in Tab. 3 compare against other open VLMs. Neither of them isolates calibration. It is hard to attribute the performance gains
-
In the controlled experiment for the learner-relative claim, the pattern disappears when labels from Qwen3.5-0.8B are transferred to LFM2.5-VL-450M. However, LFM is a weaker model with a different architecture. Its performance noise may overwhelm CMG differences.
-
The method's novelty appears limited. Several components, such as gradient-space diversification, two-stage generation, and Double-Critic filtering are all prior work. It is also mentioned by authors.
-
The scalability claim does not account for the computational cost. Each candidate requires rollouts from the target learner, LLM-based semantic verification calls, and critic calls. The paper provides no computational cost comparison with the baseline. It also does not evaluate the low-cost alternatives proposed in the Discussion.
Questions for authors
See weakness
6 comments
Responses to Reviewer ZjBz (2/2)
W2: Reliability of Learner-Relative Analysis (Continued)
Complexity Mixture Gain (CMG)
| Composition | Epoch 1 | Epoch 2 | Epoch 3 | Epoch 4 | Epoch 5 | Mean CMG |
|---|---|---|---|---|---|---|
| E1H3 | +0.018 → +1.165 (+1.148) | +0.560 → +0.240 (−0.320) | +0.330 → −0.018 (−0.348) | −0.115 → −0.790 (−0.675) | +0.063 → +0.105 (+0.043) | +0.171 → +0.141 (−0.030) |
| E2M1H1 | +0.150 → −1.383 (−1.533) | +0.340 → −1.283 (−1.623) | −0.088 → +0.098 (+0.185) | +0.360 → −0.880 (−1.240) | +0.065 → −1.010 (−1.075) | +0.166 → −0.892 (−1.058) |
The results reveal two observations:
- Single-level vs. mixed composition. With transferred labels, mixed compositions fail to outperform the best single-level baseline across all five epochs. The previously observed benefits of E1H3 over H4 also become less consistent.
- Composition gains across training. Transferring labels substantially changes the CMG trends across training dynamics. For E2M1H1, positive CMG decreases from 4/5 to 1/5 epochs, with mean CMG dropping from +0.166 to −0.892.
These findings further support our original conclusion that effective complexity composition should be calibrated relative to the target learner.
W3: Novelty of Method
We thank the reviewer for raising this concern. We agree that several components build on prior work, as cited in our paper. However, we would like to clarify the distinction between the tools we employ and our main contribution.
- When introducing a synthesis framework, accounting for these well-established data factors is essential. We use established components as tools rather than claiming novelty in them. We intentionally employ well-established techniques to reliably operationalize quality (Q) and diversity (D), rather than claiming novelty in these individual components.
- Beyond Q+D to Q+D+C in end-to-end synthesis. Our method’s main contribution is explicitly incorporating learner-relative complexity (C) alongside Q and D into an end-to-end synthesis framework, extending conventional Q+D-oriented synthesis.
We believe this represents a meaningful contribution to synthetic data generation beyond simply combining existing components.
W4: Computational Cost Analysis
We thank the reviewer for this insightful comment. We agree that the additional computational cost of complexity calibration should be discussed.
To provide a reliable comparison, we empirically measure the GPU-hours required for each component of Vanilla 16K, CoRAL 1K, and CoRAL 1K w/o complexity calibration. We account for seven computational stages:
- Data Generation:
Problem Generation (27B), Solution Generation (27B) - Diversification & Quality Control:
Gradient Diversification (0.8B), Double-Critic (27B) - Complexity Calibration:
Learner Rollouts (0.8B, N=8), Semantic Verification (27B) - Training:
Fine-tuning (2 epochs)
Following our experimental setup, we use H200 GPUs for 27B components and RTX 6000 Ada GPUs for the smaller-model operations and fine-tuning. The resulting GPU-hours and estimated costs are summarized below.
| Component | Vanilla 16K | CoRAL 1K | CoRAL 1K w/o Complexity | GPU |
|---|---|---|---|---|
| Problem Generation (27B) | 0.266 | 0.129 | 0.035 | H200 |
| Solution Generation (27B) | 3.993 | 1.939 | 0.525 | H200 |
| Gradient Diversification (0.8B) | — | 1.079 | 0.292 | RTX 6000 Ada |
| Double-Critic (27B) | — | 0.288 | 0.078 | H200 |
| Learner Rollouts (0.8B, N=8) | — | 1.881 | — | RTX 6000 Ada |
| Semantic Verification (27B) | — | 0.367 | — | H200 |
| Model Fine-tuning (2 epochs) | 2.054 | 0.153 | 0.153 | RTX 6000 Ada |
| Total H200 Hours | 4.259 | 2.723 | 0.638 | |
| Total RTX 6000 Ada Hours | 2.054 | 3.113 | 0.445 | |
| Estimated Cost (USD) | $16.81 | $12.08 | $2.62 |
Cost estimates use ‘RunPod Community Cloud’ pricing (November 2026): H200 at USD 3.59/hour and RTX 6000 Ada at USD 0.74/hour.
The results show that scaling up vanilla synthesis incurs higher costs primarily from 27B solution generation and model fine-tuning due to data scale. Although complexity calibration incurs additional computational costs through iterative processing, we consider this inference-time overhead a reasonable trade-off for generating high-quality synthetic data, consistent with prior studies that invest additional computation in data synthesis [1, 2].
—
The low-cost alternative in §Discussion was intended as a potential future direction rather than a validated method. We believe that a direct empirical comparison of computational efficiency provides more meaningful insights, and have therefore revised the §Discussion (lines 843-866) accordingly. We hope this clarification addresses the reviewer's concern.
Reference:
[1] Davidson et al., 2026. Reasoning-Driven Synthetic Data Generation and Evaluation.
[2] Jung et al., 2026. DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation.
Follow up with reviewer ZjBz
Dear Reviewer ZjBz,
We sincerely appreciate your thoughtful and constructive feedback. We provided a detailed response addressing each of your concerns, including comprehensive component-wise ablations, additional controlled experiments on learner-relative complexity calibration, clarifications of our methodological contributions, and a detailed computational cost analysis.
Given the approaching deadline, we remain fully committed to addressing any remaining questions you may have and would be grateful for any further feedback on these additional analyses.
Thank you again for your time and valuable suggestions, which have significantly strengthened our work.
Responses to Reviewer ZjBz (1/2)
We sincerely thank the reviewer for the positive evaluation of our work and the thoughtful feedback! We appreciate their recognition of complexity as an overlooked yet important factor in synthetic data design, our extensive controlled experiments, and the potential value of our released datasets to the research community.
The reviewer’s critiques have helped us clarify our methodological contributions and further strengthen the empirical evidence supporting our work.
Below, we respond to each concern with targeted analyses, supplementary experiments, and clarifications.
W1: Contribution of Individual CoRAL Components
We are grateful to the reviewer for this insightful suggestion, which motivated us to conduct additional component-wise ablations to disentangle the contributions of each components.
Beyond the requested ablation, we further evaluate all possible combinations of
Following the original vanilla synthesis comparison, we control for data scale and component combinations, and report the performance gain (Δ) from adding complexity calibration (C), as shown below.
For a clearer comparison, we encourage the reviewer to refer to Fig. 10 in the revision.
Our ablation results reveal the following findings.
Overall, these results confirm the consistent incremental contribution of complexity calibration beyond quality and diversity.
We have incorporated these component-wise ablation results to §B.4 (lines 1986–1992) and included a reference in the main text (lines 477–479).
Reference:
[1] Davidson et al., 2026. Reasoning-Driven Synthetic Data Generation and Evaluation.
W2: Reliability of Learner-Relative Analysis
We thank the reviewer for this valuable comment. Our original analysis intentionally examined learner-relative effects across models with different capacities and architectures to better reveal the importance of learner-specific complexity calibration. We also recognize the value of comparing self-assigned and transferred labels under a fixed target model to further validate these findings. We therefore conduct an additional controlled experiment.
We compare single-complexity-level performance and CMG across five representative compositions, using InternVL3.5-1B, for which a learner-specific correctness datastore is already available, given time constraints during the rebuttal period.
Below is the results. Each cell reports Self-label (i.e., learner-specific) → Transferred-label (labels transferred from Qwen3.5-0.8B).
OOD Performance in General-VQA.
W2 follow in our second comment.