acceptodds
Withdrawn from ICLR 2027

When Should an AI Tutor Withhold Feedback? Selective Instruction under Diagnostic Uncertainty

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Effective pedagogical AI systems must guide students to correct errors without prematurely revealing solutions. However, this objective is challenging when an error stems from multiple plausible causes, where even a pedagogically accurate response risks inadvertently leaking the answer. To address this, we propose PARS-Tutor, a tutoring policy that maintains multiple candidate error hypotheses and systematically suppresses feedback that fails pedagogical or safety constraints. Evaluated on 100 held-out mathematics problems, PARS-Tutor with a budget of seven hypotheses achieved a 64% success rate in generating valid, non-revealing feedback across all scheduled cases (versus 60% for direct prompting and 59% for chain-of-thought prompting); among delivered responses, its success rate reached 67.4% (versus 62.1% and 60.0%, respectively), with zero detected answer leakage across all 71 returned outputs. Furthermore, scaling the hypothesis budget from five to seven improved candidate availability by 27 percentage points, explicitly highlighting the trade-off between safety and abstention. Our findings demonstrate that post-hoc verification formalizes the tension between safety and response availability, providing a grounded framework for reliable AI tutoring while pointing to future work on downstream learning gains.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
AI tutoring, Large language models, Selective prediction, Diagnostic uncertainty, Pedagogical feedback

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_wdLvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at AI tutors that give feedback on a student’s worked math solution. The authors argue that when the tutor is unsure where the student went wrong, it should be allowed to stay silent rather than risk a hint that is wrong or gives away the answer. Their method, PARS-Tutor, considers several possible error locations, writes one hint for each, removes hints that contain numbers or calculations, and only shows the best remaining hint if a second LLM approves it. On 100 test problems, PARS-Tutor produces an acceptable hint in 64 cases, compared with 60 for a direct prompt and 59 for chain-of-thought. The hints it does show never give away the answer according to the LLM checker, while simple prompting without the filter gives it away in about a third of the cases.

Critical strengths

Letting the tutor stay silent, and reporting it openly, is a good idea. The paper reports separately how often the tutor finds the error, how often it answers at all, and how often its answers are safe, instead of mixing these into one number. The motivation from the learning sciences is convincing: a hint that gives away the answer takes away the student’s chance to work it out. The authors are also open about the limits of their own evaluation.

Critical weaknesses
  1. The improvement is small and does not seem to come from the main idea. With the same budget and the same filters, PARS-Tutor gets 64 acceptable hints out of 100, against 60 and 59 for the two simple baselines. The paper says the difference to chain-of-thought is not significant; as far as I can tell, the difference to the direct baseline is not significant either. The part of the method that handles uncertainty about the error does very little. The error finder (TF-IDF with logistic regression) picks the right step in fewer than half of the cases, and its sets of candidate steps end up containing almost every step of the solution. The authors’ own ablation shows that the drop in answer leakage comes from the simple rule that removes hints with numbers in them. That no returned hint leaks the answer is also guaranteed by design, because a hint is only shown if the checker says it does not leak. To support the main claim, the paper would need a stronger error finder, baselines that are told not to reveal the answer and use the same filter, and correct statistics.

  2. When unsure, a good tutor asks a question, and the hints are never tested with people. The tutor can only give a hint or stay silent. A human tutor who is unsure where a student went wrong would simply ask the student to explain the step. This option is not considered. Whether the hints are any good is judged only by another LLM, which is told where the error really is. There is no human rating and no measure of learning, as the authors acknowledge. Because every hint containing a calculation is removed, the remaining hints tend to be vague (“Check the multiplication in step 2”). The study is also limited to math and to a single turn and dataset. Together, this does not support the claim of “a grounded framework for reliable AI tutoring”. Finally, it was not clear to me whether the main contribution is the problem formulation, the tutoring policy or a general framework.

Questions for authors
  1. What is the VTG dataset? It is used throughout but never introduced or cited. If it is the dataset of Daheim et al. (EMNLP 2024), that paper also checks student solutions before generating feedback and should be discussed.
  2. Which models were used where? Section 5 names glm-5.2, while Figure 1(b) and Section 7 name gpt-5.6-sol as the main model. Please give exact versions and references for all models.
  3. How do the direct and chain-of-thought baselines perform when they are told not to reveal the answer and use the same filter?
  4. Could the tutor ask the student a question instead of staying silent, and how would that compare?

Minor:

  • The conclusion says the results were confirmed by “three gold-blind human judges”, but these judges are LLMs.
  • The abstract mixes numbers from two different experiments.
  • Results are given in the introduction before the method and the data are explained.
Reviewer_NLrVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper formulates math tutoring as selective feedback: keep several first-error candidates, generate a hint for the top-ranked one, pass it through a text veto and a verifier, and abstain if it fails. On 100 problems at budget 7, PARS obtains 64 strict passes against 60 for Direct and 59 for CoT, with zero detected leakage among the 71 returned outputs.

Critical strengths
  1. The formulation is clear: diagnosis, availability, and safety are separated into three estimands.
  2. The safety contrast is large: 0/71 against 36/96 and 35/99 under the same backbone and budget.
Critical weaknesses

There is a serious problem with the writing: the paper reads more like a lab notebook than a scientific study. What it lacks is analysis of the results. For example, under what conditions does a hint leak? The main text does not explain this.

  1. The headline result is not consistent across the paper's own tables. At budget 7 the strict pass count appears as 67, 64, and 71; at budget 5 it appears as 42 and 44. The term strict pass is not defined in the main text. The same backbone moves from 95.0% to 56.0% when the rubric is recalibrated, while the headline effect is 4 to 5 points.

  2. The paper does not test when the tutor should withhold; it tests whether a valid hint can be produced. Page 3 calls abstention "an active teaching decision", while page 6 calls it "a safety fallback". Four of the five abstention causes are generation, verification, or parsing failures. No human teacher annotation and no learning outcome separate "should not speak" from "cannot speak".

Questions for authors
  1. How is strict pass defined, and why is it 71, 67, and 64 at budget 7 in Tables 1, 2, and 3?
  2. At matched budget, what does a single-candidate baseline with the same veto and verifier achieve?
  3. Can the four pedagogical dimensions be re-judged gold-blind?
Reviewer_RMS6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The manuscript addresses the problem of overt tutor response generation, focusing on developing tutoring systems that can withhold feedback when it hinders the student's learning journey (e.g., due to answer leakage or misleading information). The authors formulate their proposed solution framework as a selective prediction problem, with abstention from response generation as an explicit action. Overall, I have significant concerns about the paper's writing and scientific rigor and have assigned the scores accordingly.

Critical strengths
  • Focusing on abstention as an explicit action in a tutoring interaction is an understudied but significant problem to investigate within the AI for Education community. The problem formulation (Section 3) is a valuable contribution to rethinking the problem within the community, even though some areas require a bit more specificity (see weaknesses).
Critical weaknesses
  • The paper's writing is sometimes hard to follow, with a lot of missing citations and underspecified motivations. The paper does not cite or explain what the "VTG" dataset with 1002 instances is. Neither does it cite LLMs such as the Qwen-3.5-9B model used in the analysis. Additionally, several phrases and writing excertps within the manuscript appear to be AI-slop style writing, making it hard to understand the point or method being conveyed; such as:
  • L304: "Each condition searches over N = 5 or N = 7 candidates, uses the text veto and quality ranking, and then calls feedback_verifier_calibrated"
  • L235: "If every candidate fails H, we record all_candidates_rejected_by_shield. If generation produces no usable text, we record no_valid_candidate. If the selected candidate fails the verifier, we record verifier_rejected. "
  • If the VTG dataset is indeed the work verify-then-generate, then the authors should specify that it is working on a synthetic student interaction dataset, which significantly undermines the key contribution of this work, as it suggests the efficacy of the proposed methodology on instances generated by InstructGPT.
  • The main claim of a 64% success rate of PARS-Tutor against 60% for direct prompting is computed on a small sample size of 100 instances; and given the quantitative nature of the analysis, it does not conclude with statistical significance that the proposed approach outperforms the baselines.
Questions for authors
  • What is the point the ethical statements passage is trying to convey?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.