- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper looks at AI tutors that give feedback on a student’s worked math solution. The authors argue that when the tutor is unsure where the student went wrong, it should be allowed to stay silent rather than risk a hint that is wrong or gives away the answer. Their method, PARS-Tutor, considers several possible error locations, writes one hint for each, removes hints that contain numbers or calculations, and only shows the best remaining hint if a second LLM approves it. On 100 test problems, PARS-Tutor produces an acceptable hint in 64 cases, compared with 60 for a direct prompt and 59 for chain-of-thought. The hints it does show never give away the answer according to the LLM checker, while simple prompting without the filter gives it away in about a third of the cases.
Critical strengths
Letting the tutor stay silent, and reporting it openly, is a good idea. The paper reports separately how often the tutor finds the error, how often it answers at all, and how often its answers are safe, instead of mixing these into one number. The motivation from the learning sciences is convincing: a hint that gives away the answer takes away the student’s chance to work it out. The authors are also open about the limits of their own evaluation.
Critical weaknesses
-
The improvement is small and does not seem to come from the main idea. With the same budget and the same filters, PARS-Tutor gets 64 acceptable hints out of 100, against 60 and 59 for the two simple baselines. The paper says the difference to chain-of-thought is not significant; as far as I can tell, the difference to the direct baseline is not significant either. The part of the method that handles uncertainty about the error does very little. The error finder (TF-IDF with logistic regression) picks the right step in fewer than half of the cases, and its sets of candidate steps end up containing almost every step of the solution. The authors’ own ablation shows that the drop in answer leakage comes from the simple rule that removes hints with numbers in them. That no returned hint leaks the answer is also guaranteed by design, because a hint is only shown if the checker says it does not leak. To support the main claim, the paper would need a stronger error finder, baselines that are told not to reveal the answer and use the same filter, and correct statistics.
-
When unsure, a good tutor asks a question, and the hints are never tested with people. The tutor can only give a hint or stay silent. A human tutor who is unsure where a student went wrong would simply ask the student to explain the step. This option is not considered. Whether the hints are any good is judged only by another LLM, which is told where the error really is. There is no human rating and no measure of learning, as the authors acknowledge. Because every hint containing a calculation is removed, the remaining hints tend to be vague (“Check the multiplication in step 2”). The study is also limited to math and to a single turn and dataset. Together, this does not support the claim of “a grounded framework for reliable AI tutoring”. Finally, it was not clear to me whether the main contribution is the problem formulation, the tutoring policy or a general framework.
Questions for authors
- What is the VTG dataset? It is used throughout but never introduced or cited. If it is the dataset of Daheim et al. (EMNLP 2024), that paper also checks student solutions before generating feedback and should be discussed.
- Which models were used where? Section 5 names glm-5.2, while Figure 1(b) and Section 7 name gpt-5.6-sol as the main model. Please give exact versions and references for all models.
- How do the direct and chain-of-thought baselines perform when they are told not to reveal the answer and use the same filter?
- Could the tutor ask the student a question instead of staying silent, and how would that compare?
Minor:
- The conclusion says the results were confirmed by “three gold-blind human judges”, but these judges are LLMs.
- The abstract mixes numbers from two different experiments.
- Results are given in the introduction before the method and the data are explained.