- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces Sampled Pragmatic Adaptation (SPA), a method that aims to improve tool invocation by reasoning about alternative user meanings and selecting the most informative responses. Experiments with a single small-scale LLM (Qwen3-0.6B) show mixed results compared to the baseline. Overall, I do not think the paper is ready for publication given the current empirical evidence.
Critical strengths
I appreciate the idea of introducing RSA-style pragmatic reasoning into LLM-based agentic workflows (specifically, tool invocation in this work). The proposed SPA generates alternative meanings using LLMs rather than relying on fixed or predefined alternatives, providing greater flexibility.
Critical weaknesses
The biggest concern is the empirical support for SPA. The experiments do not demonstrate consistent improvements over the basic literal baseline, and the error and failure analyses do not sufficiently explain when SPA performs better or worse than the baseline. A more informative analysis of the domains and task types where SPA succeeds or struggles would help clarify its effectiveness.
Furthermore, the experiments are limited to a single small-scale model, Qwen3-0.6B. Larger reasoning LLMs may already be capable of understanding user intents without an explicit pragmatic reasoning framework. Therefore, the current findings are not sufficiently convincing or generalizable without further empirical evidence.
A key component of SPA is the generation of alternative user intents given the conversational context, for which the prompt is provided in Appendix C. A standalone evaluation of the generated intents, including the extent to which they deviate from the observed intent, would help assess the reliability of this step in disambiguating user intents.
The additional API costs, inference time, and memory usage introduced by SPA should also be reported to enable a fair comparison with the baseline.
Minor: The font sizes in the figures are too small, making them difficult to read.
Questions for authors
The motivation for the marginal-matching constraint is not sufficiently clear. Specifically, the authors require the average adapted response distribution over sampled alternative intents to match the unadapted response distribution of the original query. However, the unadapted model does not explicitly reason about alternative intents and may produce biased or poorly calibrated response distributions. It is therefore unclear why this distribution should serve as a reliable target marginal for pragmatic adaptation, or why enforcing this constraint should improve downstream task performance. A stronger justification or an empirical comparison with alternative marginal choices would help clarify this design decision.