- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents DeepTutor, a system that can be used both for guiding students in tutoring sessions and for generating problems for these students. Both of these capabilities make use of the same knowledge bases consisting of course evidence and a student profile that is consistently updated as interactions progress. The paper also presents a benchmark called Tutorbench that is used to evaluate DeepTutor on 270 tasks with, as the authors report, improvements over baseline systems.
Critical strengths
-
The paper is very easy to read and follow.
-
All methods that are used make sense and contribute towards meaningful pedagogical criteria that the authors have outlined.
Critical weaknesses
-
Overall, it is rather unclear what is to gain from this paper, as it mainly seems to be taking a collection of established ideas (like RAG, scratchpad-like memory that evolves over time, rubrics, etc.) and applies them to tutoring. The execution seems sound as far as it is possible to assess with the details given (see below) but it is unclear what is to gain scientifically beyond what is already known from prior work.
-
Many details are left out, for example, how exactly DeepTutor is implemented (not in terms of code but, for example, whether it is implemented with the same model but different prompts or different models). Also, TutorBench is not really described in the main paper so it is unclear what it actually does and how the quality of the data is.
Questions for authors
Questions:
-
Can you please more clearly define what DeepTutor is? Is it an agentic system made up of components like planner, tool agent, etc. (see e.g. Sec. 3.2, 3.3) that are instantiated via different prompts or are these different models? I could not find these details, also in Appendix B. Is it correct to interpret L350 in a way that all are different Gemini instantiations?
-
Can you please provide more details on TutorBench? What are the tasks exactly? How did you ensure data quality is sufficient?
Suggestions for a next draft:
-
Fig. 2 is very hard to read due to small text and text blending together with objects, I would suggest to improve its clarity.
-
Many of the figures should also be made .pdfs
3 comments
Dear Authors,
Thank you for your response!
"We believe real-world tutoring requires two capabilities at once: general reasoning ability to handle the varied questions posed by different students across disciplines, and effective instructional design that adapts to each individual's learning process. Prior work tends to focus on only one of the two."
This is not exactly true at least for benchmarks, right? For example, the idea of MathTutorBench is to evaluate both reasoning capabilities as well as tutoring abilities. Maybe this can be discussed a bit more in the manuscript, comparing (per discussion) to other tutoring datasets would be helpful in general.
Otherwise I do not yet have any further questions and will revise my scores after also reading the other responses.
Dear Reviewer wunw,
Thank you so much for your reply and for revising your score, we really appreciate it! :)
You are right, our statement about prior work was too broad, and benchmarks like MathTutorBench (Macina et al., 2025) do evaluate both reasoning and tutoring abilities. We will tone down that claim, and discuss the positioning of TutorBench relative to other related benchmarks in our manuscript. We will also fix the layout of several figures in our paper. Thanks again for your detailed responses and timely replies!
Wish you all the best!
References
Jakub Macina, Nico Daheim, Ido Hakimi, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. MathTutorBench: A benchmark for measuring open-ended pedagogical capabilities of LLM tutors. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 204–221, 2025.
Thank You for Your Feedback and a Kind Follow-Up on Our Response
Dear Reviewer wunw,
We sincerely appreciate your detailed review and your questions about the scientific contribution, system implementation, and TutorBench. As the discussion deadline is approaching, we would be grateful if you could take another look at our response.
We clarified that DeepTutor is not merely a collection of prompts: while all components use the same Gemini backbone for controlled comparison, each component has its own tools, memory structures, state management, and role in the tutoring workflow. More importantly, the ELP is designed to revise evidence-grounded judgments about a learner across sessions, rather than simply retain a conversational scratchpad. Our new controls show that this evolving profile outperforms both length-matched summaries and retrieval-only memory, while its estimates of learners’ true knowledge gaps become substantially more accurate over time.
We also provided the complete construction and quality-control process for TutorBench. If these additions have helped address your concerns, we would sincerely appreciate your reconsideration of the rating. We remain very happy to answer any outstanding questions.