- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper trains a hint-generating teacher using a utility signal defined by the change in a frozen student’s likelihood of the correct answer. Candidate hints are screened for validity, scored for student utility, and used to construct preference data for teacher refinement. The evaluation reports improvements across eight data sources and two main student models, together with selection analyses and exploratory transfer experiments. The target is immediate assistance to a frozen student, rather than persistent learning or student parameter updates. The results support gains for this operational objective.
Critical strengths
SIMPLE makes a useful distinction between hint validity and hint utility. Grounding and anti-hacking checks restrict which candidates receive credit, while student-conditioned likelihood gains distinguish among valid hints. Coverage correction also accounts for failures to produce valid candidates. Together, these choices provide a concrete and reusable training signal for learning by teaching.
The reported gains span eight data sources and both main student models, supporting the practical value of the complete training procedure within its stated objective. The additional selection and exploratory transfer analyses help characterize the signal beyond a single aggregate result. This empirical breadth, combined with the operational clarity of the objective, supports acceptance despite the remaining attribution and generalization questions.
Critical weaknesses
The computational cost could be documented more fully. The paper reports training time, but candidate generation and student-based scoring also contribute to the overall cost. A clearer breakdown would help readers understand the practical requirements of applying SIMPLE and compare it with the simpler training baselines. This is primarily a reporting limitation rather than a concern about the observed gains.
The transfer analyses provide useful additional evidence, although their relationship to the final system could be clearer. Some experiments concern hint selection or a pre-refinement checkpoint, while the main results concern the final SIMPLE teacher. Explicitly separating these settings would make the scope of the cross-student findings easier to interpret.
Questions for authors
Could the authors provide an approximate breakdown of the computation spent on candidate generation, student-based scoring, and teacher training? It would also be helpful to indicate which stage accounts for most of the additional cost relative to the baselines.
Could the authors clarify which cross-student results apply to the final SIMPLE teacher and which apply to the pre-refinement checkpoint or selection procedure? If the final teacher has been evaluated on additional students, a brief summary would help readers understand its transferability.
4 comments
Follow-up on Our Rebuttal Responses
Dear Reviewer k7v7,
I hope this message finds you well. As the discussion period is nearing its end with less than two days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you’d like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.
Thank you for your time and effort in reviewing our paper.
Thank you for the response, which addressed most of my concerns. I believe this is a good paper, so I will maintain my positive assessment.
Thank You for Your Constructive Feedback
Dear Reviewer k7v7,
Thank you for your careful review and continued positive assessment of our work. Your questions helped us provide a more complete account of the computational costs and clarify the checkpoints and student roles in the cross-student evaluations.
We appreciate your constructive guidance and will incorporate these clarifications into the revised manuscript to make the practical requirements and scope of our findings clearer.
Response to Reviewer k7v7
We thank the reviewer for the positive assessment of SIMPLE's operational objective and empirical results, and for identifying two areas where the reporting can be clearer. Below, we provide an approximate cost breakdown for the Qwen SIMPLE training pipeline, including data preparation, and clarify the scope of the cross-student evaluations.
Thank you for pointing out this reporting gap. We profiled candidate generation and student scoring separately and combined these estimates with the recorded teacher-training times. The approximate breakdown is:
The total is computed from unrounded values, so it differs slightly from the sum of the displayed entries. Data-preparation costs are estimated by profiling 32 examples per collection stage and extrapolating to the corresponding 5,000-example collections. They are not exact historical timings. The 4.0 GPU-hours correspond to 60.2 minutes of teacher training on four NVIDIA RTX 3090 GPUs, comprising 32.8 minutes for Hint-SFT and 27.4 minutes for refinement.
The initial data preparation is shared with Hint-SFT. Relative to this baseline, the additional refinement-data collection costs approximately 15.1 GPU-hours, compared with 1.8 GPU-hours for refinement training. Data collection is therefore the main source of SIMPLE's additional cost. We will include this breakdown, the hardware configuration, and the estimation procedure in the revision.
Thank you for highlighting this ambiguity. The labels "pre-refinement SimPO" and "pre-RFT SimPO" were incorrectly retained in the manuscript. The cross-student analyses in Figures 7-8 and Table 5 use the final Qwen3-1.7B SIMPLE checkpoint reported in Table 1, trained through Hint-SFT followed by reward-filtered preference refinement. This checkpoint is distinct from the SimPO baseline in Table 1. The exploratory two-student refinement checkpoint in Appendix L is a separate experiment. We will correct the legacy labels consistently and explicitly identify the teacher checkpoint, selection student(s), and evaluation student for each analysis.
For example, on 300 SQuAD events, the final teacher achieves coverage-corrected masked utility of 0.368 for Qwen3-0.6B, with Qwen3-1.7B selecting the hints. Additional accuracy evaluations using the same teacher and student roles show improvements of 0.20-1.01 percentage points over the no-hint condition across SciQ, ARC-Easy, ARC-Challenge, RACE-Middle, and RACE-High.
We will also distinguish exclusion from hint selection from exclusion from training feedback. Qwen3-0.6B contributed feedback to initial Hint-SFT data construction, although it did not contribute to subsequent refinement feedback or hint selection in these evaluations. These Qwen3-0.6B results therefore support cross-student usefulness, with the qualification that this student was not unseen throughout teacher training.