acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Agentic Tutoring with Evolving Learner Profiles

Abstract

Tutoring a learner through a question and choosing what they should practice next require different operations, but both depend on the same course material and evidence of that learner’s difficulties. We present DeepTutor, which separates problem tutoring from practice generation while sharing course retrieval and an evolving learner profile revised from interaction traces. The tutoring pipeline investigates, solves with tools, and writes an evidence-backed explanation; the practice pipeline selects a learning target before generating and validating question–answer–explanation items. We introduce TutorBench to evaluate both outputs in multi-turn dialogue: 270 tasks across five disciplines pair source-grounded knowledge gaps with 90 benchmark learner specifications, and a first-person student simulator elicits tutoring and follow-up practice. Experiments on TutorBench show that DeepTutor attains the highest average scores for both tutoring and generated practice. Ablations show that course grounding helps explanations and practice stay faithful to source material, while the evolving learner profile helps tailor them to learners’ difficulties. In a blind comparison with an agentic tutoring baseline, human raters and the LLM judge both favor DeepTutor across all ten evaluation dimensions. These results support coordinating distinct tutoring and practice pipelines through shared course evidence and an evolving learner profile.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Agentic Tutoring, Personalized Learning, Learner Modeling, Practice Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wunwOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents DeepTutor, a system that can be used both for guiding students in tutoring sessions and for generating problems for these students. Both of these capabilities make use of the same knowledge bases consisting of course evidence and a student profile that is consistently updated as interactions progress. The paper also presents a benchmark called Tutorbench that is used to evaluate DeepTutor on 270 tasks with, as the authors report, improvements over baseline systems.

Critical strengths
  • The paper is very easy to read and follow.

  • All methods that are used make sense and contribute towards meaningful pedagogical criteria that the authors have outlined.

Critical weaknesses
  • Overall, it is rather unclear what is to gain from this paper, as it mainly seems to be taking a collection of established ideas (like RAG, scratchpad-like memory that evolves over time, rubrics, etc.) and applies them to tutoring. The execution seems sound as far as it is possible to assess with the details given (see below) but it is unclear what is to gain scientifically beyond what is already known from prior work.

  • Many details are left out, for example, how exactly DeepTutor is implemented (not in terms of code but, for example, whether it is implemented with the same model but different prompts or different models). Also, TutorBench is not really described in the main paper so it is unclear what it actually does and how the quality of the data is.

Questions for authors

Questions:

  • Can you please more clearly define what DeepTutor is? Is it an agentic system made up of components like planner, tool agent, etc. (see e.g. Sec. 3.2, 3.3) that are instantiated via different prompts or are these different models? I could not find these details, also in Appendix B. Is it correct to interpret L350 in a way that all are different Gemini instantiations?

  • Can you please provide more details on TutorBench? What are the tasks exactly? How did you ensure data quality is sufficient?

Suggestions for a next draft:

  • Fig. 2 is very hard to read due to small text and text blending together with objects, I would suggest to improve its clarity.

  • Many of the figures should also be made .pdfs

3 comments
AuthorsOpenReview ↗

Thank You for Your Feedback and a Kind Follow-Up on Our Response

Dear Reviewer wunw,

We sincerely appreciate your detailed review and your questions about the scientific contribution, system implementation, and TutorBench. As the discussion deadline is approaching, we would be grateful if you could take another look at our response.

We clarified that DeepTutor is not merely a collection of prompts: while all components use the same Gemini backbone for controlled comparison, each component has its own tools, memory structures, state management, and role in the tutoring workflow. More importantly, the ELP is designed to revise evidence-grounded judgments about a learner across sessions, rather than simply retain a conversational scratchpad. Our new controls show that this evolving profile outperforms both length-matched summaries and retrieval-only memory, while its estimates of learners’ true knowledge gaps become substantially more accurate over time.

We also provided the complete construction and quality-control process for TutorBench. If these additions have helped address your concerns, we would sincerely appreciate your reconsideration of the rating. We remain very happy to answer any outstanding questions.

Dear Authors,

Thank you for your response!

"We believe real-world tutoring requires two capabilities at once: general reasoning ability to handle the varied questions posed by different students across disciplines, and effective instructional design that adapts to each individual's learning process. Prior work tends to focus on only one of the two."

This is not exactly true at least for benchmarks, right? For example, the idea of MathTutorBench is to evaluate both reasoning capabilities as well as tutoring abilities. Maybe this can be discussed a bit more in the manuscript, comparing (per discussion) to other tutoring datasets would be helpful in general.

Otherwise I do not yet have any further questions and will revise my scores after also reading the other responses.

Dear Reviewer wunw,

Thank you so much for your reply and for revising your score, we really appreciate it! :)

You are right, our statement about prior work was too broad, and benchmarks like MathTutorBench (Macina et al., 2025) do evaluate both reasoning and tutoring abilities. We will tone down that claim, and discuss the positioning of TutorBench relative to other related benchmarks in our manuscript. We will also fix the layout of several figures in our paper. Thanks again for your detailed responses and timely replies!

Wish you all the best!

References

Jakub Macina, Nico Daheim, Ido Hakimi, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. MathTutorBench: A benchmark for measuring open-ended pedagogical capabilities of LLM tutors. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 204–221, 2025.

Reviewer_EguEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DeepTutor separates problem tutoring from practice generation while sharing course retrieval and an evolving learner profile. A Profile Agent updates session summaries, knowledge-gap hypotheses, and pedagogical reflections from searchable hierarchical traces. Tutoring follows an investigate–solve–write workflow; practice generation selects targets and validates question–answer–explanation items. TutorBench contains 270 tasks from 90 learner specifications and 30 knowledge bases across five disciplines, evaluated through simulated conversations and ten rubric dimensions. DeepTutor improves average rubric scores over backbone-matched prompting controls. Ablations associate course grounding with source fidelity and profile/memory removal with reduced personalization. A blind human comparison with GenMentor favors DeepTutor. The paper evaluates output quality, not subsequent learner achievement.

Critical strengths
  1. Coherent system design for two connected educational tasks. Sections 3.1–3.4 give a concrete way for evidence from tutoring to affect later practice generation. The Trace Forest preserves the provenance of revisable learner hypotheses, and the text correctly avoids equating generated practice with observed student mastery.
  2. Broader evaluation than a single model-judge score. The 270-task, five-domain evaluation includes backbone-matched controls, SKG/ELP ablations, rubric-composition sensitivity, and a blinded two-rater comparison across all tasks. Explicitly describing the human/LLM correlation as a correlation of ten metric-level win rates, rather than item-level agreement, is appropriate. The paper also acknowledges low absolute practice-groundedness scores.
Critical weaknesses
  1. The central benefit of evolving learner profiles is not distinguished from access to memory and extra learner context. Section 5.2 and Appendix C.3 remove the profile, profile updates, and access to prior traces together. This establishes that the combined information channel helps, but not that revising a profile is better than a static initial profile, a running conversation summary, or direct retrieval of the same history. These alternatives matter because evolution is the title-level contribution. Appendices B/C describe the mechanics, but the experiments do not quantify whether gap hypotheses become more accurate, whether revisions correct earlier errors, or whether later practice improves as a result of a specific update. The contribution currently reads as a plausible integrated memory system whose distinctive mechanism is not isolated.
  2. The evaluation does not fully separate personalized pedagogical quality from resource and presentation effects. The backbone-matched controls share a model and task interface, but not the multi-agent retrieval, planning, solving, writing, and validation computation; the limitations explicitly acknowledge the lack of cost matching. Vividness and Logical Depth account for 39–46% of the gap to prompting controls. Removing those metrics leaves a positive gap, which is useful, but does not control response length or extra tool use. The human comparison only uses GenMentor, not the strongest overall independent competitor, Claude Code (OQ 3.78 versus 3.91; practice 3.38 versus 3.42), and no learner/KB-cluster uncertainty accompanies these small aggregate gaps. Thus there is evidence for better rated outputs, but weaker evidence for the claimed benefit specifically attributable to adaptive learner modeling. I am not asking the paper to establish real learning gains that it explicitly disclaims.
Questions for authors
  1. Existing evidence: Are static-profile, running-summary, or history-retrieval controls already available? If not, which existing trace examples or profile-update logs show a corrected learner hypothesis affecting later practice?
  2. Clarification using recorded logs: What are response lengths, interaction turns, model/tool calls, and available costs for DeepTutor and matched controls? Are the gains concentrated in longer interactions?
  3. Existing-data clarification: Are paired learner- or KB-level uncertainty estimates and raw human agreement rates available? The correlation across ten metric means does not answer those questions. New human learning trials are future work, not a rebuttal requirement.
Reviewer_T876OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents DeepTutor, an agentic tutoring system that coordinates problem tutoring and practice generation through shared course retrieval and an evolving learner profile. A ProfileAgent revises session summaries, knowledge-gap hypotheses, and pedagogical reflections using searchable interaction traces organized as a Trace Forest. The paper also introduces TutorBench, comprising 270 tasks associated with 90 synthetic learner specifications and 30 knowledge bases across five disciplines. Evaluation combines simulated student interactions, rubric-based LLM scoring, component ablations, rubric-sensitivity analyses, and a blind human comparison against GenMentor. DeepTutor achieves the highest aggregate tutoring and practice scores among the evaluated systems, although the experiments do not fully isolate the contribution of profile evolution from simpler learner-memory mechanisms.

Critical strengths
  1. The paper presents a coherent framework that connects tutoring and practice generation through shared course evidence and an updatable learner profile.
  2. The evaluation is comprehensive, including matched-backbone comparisons, component ablations, cross-domain experiments, and a blind human evaluation across 270 tasks.
Critical weaknesses
  1. The ELP ablation removes the learner profile, its updates, and historical traces simultaneously, making it difficult to isolate the contribution of profile evolution. Comparisons with static profiles or dialogue summaries, along with direct evaluation of profile-update accuracy, are missing.

  2. The evaluation does not account for differences in inference cost and response length, and lacks uncertainty estimates. The approximately 0.12-point advantage over Claude Code is particularly difficult to assess, as Claude Code is excluded from the human evaluation.

Questions for authors
  1. How is the initial learner profile constructed? How are tasks ordered, when are profiles and traces reset, and what conversation history is available to each baseline?
  2. Does the proposed profile evolution and trace retrieval outperform a simple summary-based memory baseline under a comparable inference budget?
  3. Can the authors report confidence intervals accounting for shared learners or knowledge bases, along with instance-level human–LLM agreement and inter-annotator agreement?
6 comments
AuthorsOpenReview ↗

A Kind Follow-Up on the New Experiments Suggested in Your Review

Dear Reviewer T876,

Thank you again for your thoughtful and highly specific feedback. As the discussion deadline is approaching, we would be sincerely grateful if you could revisit our response.

Following your suggestions, we added the static-profile and rolling-summary controls, directly evaluated profile-update accuracy across sessions, reported learner-clustered confidence intervals, and provided item-level human–LLM and inter-annotator agreement.

In particular, the full ELP consistently outperforms both summary memory and retrieval-only history under comparable settings, while its profile recall improves from 0.47 to 0.84 and precision from 0.58 to 0.81. We believe these additions directly address your main concerns about isolating profile evolution and strengthening the evaluation.

If you feel that these concerns have been sufficiently resolved, we would sincerely appreciate your reconsideration of the assessment. If anything remains unclear, we would be very happy to provide further clarification.

Dear Reviewer T876,

Just a quick note before the discussion period ends. In response to your concerns and questions, we added summary-memory and frozen-profile controls, a profile accuracy analysis across sessions, learner-level confidence intervals, and item-level human–LLM judge agreement, and we clarified how each system handles profiles and history. We've also presented all the six follow-up experiments in our general response.

We would be grateful if you could take a look. If our responses address your concerns, we would appreciate it if you could consider revising your score; if not, please let us know what remains unclear and we will be glad to discuss it before the deadline of the discussion period.

Thanks for your time, and we are looking forward to your replies! :)

Best regard, Authors.

Reviewer_T876OpenReview ↗

Response to author

Thank you for the additional experiments. I still have some questions about how DeepTutor's end-to-end inference cost compares with that of Claude Code. However, the new comparisons against a rolling summary and a frozen profile address my main concern regarding whether updating the learner profile actually improves performance on TutorBench. While some questions remain, this resolves my primary concern, and I am willing to reconsider my rating.

Comments on the paper

2 comments

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,

Area Chair

Dear Area Chair,

Thank you for coordinating the discussion and for encouraging the reviewers to engage with our rebuttal. We sincerely appreciate your time and effort. We remain happy to answer any further questions from the reviewers during the remaining discussion period.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.