- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Dual Tuning jointly assesses the utility of candidate training data and supervision preferences across task groups through a single multimodal fine-tuning run using both chain-of-thought (CoT) and direct-answer (DA) supervision. By combining performance gains over the base model with performance differences between the CoT and DA modes, the method provides a basis for data selection and supervision assignment. Experiments across tasks and models show that the screening outcomes depend on the specific training configuration, while subset retraining provides directional evidence supporting the use of these signals for subsequent data curation.
Critical strengths
The paper presents several interesting findings, suggesting that the advantages of CoT are strongly associated with task characteristics, data quality, and base model capabilities. On many spatial perception tasks, such as object counting and size estimation, as well as some multidisciplinary tasks, DA outperforms CoT while requiring less computation. These findings suggest that selecting appropriate supervision formats for different tasks may offer opportunities to improve model performance during post-training.
Critical weaknesses
The role of CoT data quality remains unclear. Could the performance degradation observed after training with CoT data be attributable to a distribution mismatch between the CoT data and the base model? It is also unclear how sensitive the experimental results are to differences in CoT data quality.
The transferability of screening outcomes across base models has not been sufficiently validated. The model comparisons do not cover different scales or versions within the same model family, leaving it unclear whether task-group supervision preferences and data selection decisions can be reused across such models.
Questions for authors
I would appreciate a detailed clarification of the following two issues:
-
Changing the source of CoT data affects the screening outcomes, but it remains unclear whether these changes primarily arise from differences in reasoning trace quality or distribution mismatches between the teacher and the base model. Please clarify whether comparisons controlling for other variables have been conducted, and provide information on reasoning correctness and trace length to help explain the negative gains observed in the CoT mode on some tasks.
-
Subset retraining provides directional evidence supporting the screening signals. However, the performance of a final model trained on the combined selected data, using the recommended supervision format for each task group, has not yet been evaluated.
I also consider the following question a worthwhile direction for further investigation:
- Different base models may exhibit different supervision preferences, but it remains unclear whether screening outcomes can be reused across models of different scales or versions within the same family. Please discuss this transferability and clarify whether experiments have examined whether using one model's screening outcomes to guide data selection and supervision assignment for another model preserves the corresponding performance gains.
Thanks for the clarification, but the paper still lacks a direct experimental comparison with existing data selection methods, so I will keep my original score.