- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors argue that over-teaching can cause harm to the student model, and present FAR, a pipeline that targets the weakness of a student model, which first identifies what is missing and curates a targeted lesson to teach the student model to learn those missing capabilities.
Critical strengths
Overall, the idea of targeting what capability is still missing in the current student checkpoint is sound and sensible.
Critical weaknesses
-
There are a lot of design choices in the pipeline, and few of them are validated by experiments. For example, in line 203 "We summarize each solution by frequency-weighted pooling (Arora et al., 2017), using IDF so terms repeated across correct solutions contribute less", it's not clear whether this is the best way to do so. Same for line 213, "the lesson count is its rounded effective rank (Roy & Vetterli, 2007)..." and line 221 "the over-teaching rate".
-
The main weakness of this paper is that the student models are too weak. The authors use Mistral-7B-Instruct-v0.2 and Qwen1.5-4B-Chat, both are outdated. It is understandable to train small models, but it is necessary to show the effectiveness of the proposed pipeline on models later than Qwen3 / Gemma 3.
-
Relatedly, many of the benchmarks evaluated are saturated nowadays with Qwen3-4B-Instruct-2507 or newer models.
Questions for authors
-
The center claim "Repeated correct solutions can over-teach the student while leaving questions unsolved" is not intuitive to me. Like SFT or rejection sampling fine-tuning, using multiple correct solutions to teach the student model shouldn't hurt what the student already knows about that particular problem, and can even improve its overall performance. Is there any reference or empirical result can support this?
-
It is not very clear what "Initial", "Sustained", "Corrected" mean in Figure 2. How "State-aware geometry" separate these groups? The method section is overly complex, with many equations that might not be necessary, making it harder to follow and less connected to the figure.
-
The method seems to form a loop with Fold-Attempt-Revisit, but in the figure and text, it seems that the authors only do this loop once. What happens if we iteratively train the student model?