- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Future-Grounded Supervision, which uses real-world user behavior histories and observed future actions as supervision signals to improve LLMs' understanding of users' long-term preferences, intentions, and behavior patterns. The authors introduce Vela, a framework that converts de-identified user behavior histories into various natural language tasks, followed by difficulty filtering and teacher-generated trajectory selection. The resulting data is used to fine-tune Qwen3.6-35B-A3B, and the authors further investigate its effectiveness as an initialization for reinforcement learning.
Experiments show that Vela improves performance on the proposed VelaBench, several user-related benchmarks, and some general capability benchmarks. The paper also examines the effects of data scaling and RL initialization.
Critical strengths
-
Interesting and practically relevant problem. The paper addresses how real-world user behavior can be used to improve LLMs' understanding of users. The problem is clearly motivated and relevant to personalized AI agents.
-
Meaningful data construction effort. The authors convert real-world user behavior into diverse training tasks and develop a fairly complete data construction pipeline. However, its long-term value to the community will depend on the reproducibility and broader adoption of the data and construction process.
-
Positive empirical results. Vela achieves improvements on several user-related tasks and general benchmarks, demonstrating the value of the constructed data for LLM post-training.
Critical weaknesses
-
The contribution of Future-Grounded Supervision is not sufficiently isolated. The proposed approach mainly consists of task construction, data filtering, teacher-generated trajectories, and standard SFT, with limited methodological novelty. While the experiments demonstrate that the overall pipeline is effective, they do not establish whether Future-Grounded Supervision provides additional benefits over conventional user-domain SFT. Therefore, the evidence is not yet sufficient to support presenting it as a new supervision paradigm.
-
The independence and reliability of some evaluations need further validation. VelaBench is constructed using the same task templates as the training data. Although the training and evaluation users do not overlap, the model may still benefit from familiarity with the task formats. In addition, Qwen3.8-2.4T-A95B is used both to generate training trajectories and to evaluate some benchmarks, which may introduce evaluation bias. The authors should clarify the sources and independence of the user-related OOD benchmarks and provide validation with an independent evaluator.
Questions for authors
- What additional benefits does Future-Grounded Supervision provide over conventional user-domain SFT?
Could the authors include a controlled SFT baseline that does not use future-behavior feedback, while keeping the teacher model, training data size, and optimization settings the same? This would help establish the specific contribution of Future-Grounded Supervision.
- How much does each task type contribute to the performance gains?
The four task types (Objective, Open-prediction, Open, and Attribution) use different forms of supervision. Could the authors provide an ablation study to clarify the contribution of each type, particularly the role of actual future-behavior supervision?
- How are the independence and reliability of the user-related benchmarks ensured?
Are the four user-related OOD benchmarks existing public datasets or newly constructed by the authors? How is their independence from the training data ensured? Also, could the authors validate the main results using an evaluator different from the teacher model?
Thank you for the detailed responses
Thank you for the detailed responses and for conducting the additional experiments. I appreciate the authors' efforts to address my concerns.
The evaluation with independent judges helps alleviate my concern about potential evaluation bias, and the additional ablation provides useful evidence that future-grounded selection contributes to the observed improvements beyond teacher distillation alone. However, the improvements attributable specifically to future-grounded selection remain relatively modest compared with the overall gains, leaving some uncertainty about its contribution beyond teacher distillation.
Overall, while the additional experiments and clarifications have addressed several of my concerns, I still have some reservations regarding the strength of the central claim. I therefore intend to maintain my original rating.