acceptodds
Under review as a conference paper at ICLR 2027

Sampled Pragmatic Adaptation: Zero-shot Pragmatic Inference for Language Models

Abstract

Large language models (LLMs) may assign high probability to various responses which are plausible for a query but are also compatible with nearby interpretations of the user’s intent. We introduce Sampled Pragmatic Adaptation (SPA), a zero-shot inference-time adaptation method that use this ambiguity to improve its output distribution by generating a local neighborhood of query-specific alternative intents. We formulate the adaptation through a minimum-intervention principle: among distributions satisfying query-local marginal constraints, we seek the one that minimally departs from the original distribution in KL divergence. The resulting correction connects pragmatic reasoning under the Rational Speech Act framework, iterative proportional fitting, and prior-shift calibration. We derive utterance- and token-level SPA variants that require neither training nor task-level calibration data, making the approach applicable to open-ended sequential generation. We study the method on tool invocation, a controlled setting in which models must select and construct function calls from conversational context and tool specifications, and introduce StaTI, a static benchmark assembled from existing datasets to evaluate tool invocation independently of downstream execution.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Zero-shot adaptation, Iterative Proportional Fitting Procedure, Rational Speech Act, Prior-shift

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2B7rOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces Sampled Pragmatic Adaptation (SPA), a method that aims to improve tool invocation by reasoning about alternative user meanings and selecting the most informative responses. Experiments with a single small-scale LLM (Qwen3-0.6B) show mixed results compared to the baseline. Overall, I do not think the paper is ready for publication given the current empirical evidence.

Critical strengths

I appreciate the idea of introducing RSA-style pragmatic reasoning into LLM-based agentic workflows (specifically, tool invocation in this work). The proposed SPA generates alternative meanings using LLMs rather than relying on fixed or predefined alternatives, providing greater flexibility.

Critical weaknesses

The biggest concern is the empirical support for SPA. The experiments do not demonstrate consistent improvements over the basic literal baseline, and the error and failure analyses do not sufficiently explain when SPA performs better or worse than the baseline. A more informative analysis of the domains and task types where SPA succeeds or struggles would help clarify its effectiveness.

Furthermore, the experiments are limited to a single small-scale model, Qwen3-0.6B. Larger reasoning LLMs may already be capable of understanding user intents without an explicit pragmatic reasoning framework. Therefore, the current findings are not sufficiently convincing or generalizable without further empirical evidence.

A key component of SPA is the generation of alternative user intents given the conversational context, for which the prompt is provided in Appendix C. A standalone evaluation of the generated intents, including the extent to which they deviate from the observed intent, would help assess the reliability of this step in disambiguating user intents.

The additional API costs, inference time, and memory usage introduced by SPA should also be reported to enable a fair comparison with the baseline.

Minor: The font sizes in the figures are too small, making them difficult to read.

Questions for authors

The motivation for the marginal-matching constraint is not sufficiently clear. Specifically, the authors require the average adapted response distribution over sampled alternative intents to match the unadapted response distribution of the original query. However, the unadapted model does not explicitly reason about alternative intents and may produce biased or poorly calibrated response distributions. It is therefore unclear why this distribution should serve as a reliable target marginal for pragmatic adaptation, or why enforcing this constraint should improve downstream task performance. A stronger justification or an empirical comparison with alternative marginal choices would help clarify this design decision.

Reviewer_wJa7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Sampled Pragmatic Adaptation (SPA), a training-free inference-time distribution adaptation method for large language models. The method samples alternative user intents related to the original query and adjusts the model's output probabilities through KL projection. The authors establish theoretical connections among Rational Speech Act (RSA), Iterative Proportional Fitting (IPFP), and prior-shift calibration, and introduce both utterance-level and token-level implementations. The experiments primarily focus on tool invocation tasks.

Critical strengths

Solid theoretical foundation. The method is formulated with a well-defined KL minimization objective and establishes mathematical connections to existing approaches in pragmatic reasoning and distribution calibration.

Training-free adaptation. SPA requires neither labeled data nor additional fine-tuning. It constructs query-specific alternative intents, offering potential generality and interpretability.

Relatively comprehensive analysis. The authors provide theoretical analyses of the consistency and asymptotic normality of the sampling-based estimator and introduce the StaTI benchmark to support the evaluation of tool invocation performance.

Critical weaknesses

Limited empirical effectiveness. Although SPA improves performance on certain tool invocation tasks, it substantially reduces accuracy on others. For instance, performance deteriorates on multiple API-Bank subtasks. Overall, the experimental results do not demonstrate consistent and substantial improvements over the baseline.

Limited experimental scope. The experiments are conducted only with Qwen3-0.6B, making it difficult to assess whether the proposed method generalizes to stronger LLMs, particularly those with more advanced intent understanding capabilities.

Lack of competitive inference-time baselines. The paper does not provide sufficient comparisons with alternative approaches such as repeated sampling, candidate reranking, self-consistency, and simple probability calibration under matched computational budgets. Consequently, it is difficult to isolate the specific contribution of SPA's probability correction mechanism.

Gap between theoretical objectives and downstream performance. The KL projection guarantees a minimum-intervention property of the distributional adjustment, but it does not directly guarantee improved tool invocation accuracy. Furthermore, the influence of alternative-intent sampling quality on the final performance requires more thorough investigation.

Questions for authors

1, What factors determine whether SPA improves or degrades the accuracy of the original model? Could the authors provide a more systematic analysis of failure cases?

2, Does SPA still outperform simpler sampling and reranking approaches when evaluated with stronger language models under matched inference budgets?

3, How robust is the proposed method when the sampling distribution of alternative intents deviates from the true semantic neighborhood of the original query?

Reviewer_pcY1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Authors proposes Sampled Pragmatic Adaptation (SPA), a method to adjust LLM output non-parametrically (IE, with no training). When a user sends a message, the model first writes 8 alternative versions of that message. It then checks how likely each candidate answer is under the real message and under the alternatives. It boosts answers that fit the real message specifically over others. Authors show two versions -- one reranks 8 complete candidate answers; the other adjusts the model word by word as it writes. The paper also states to release StaTI, a derived set of 7648 tool-calling examples from BFCL, API-Bank and NoisyToolBench in one common format. Experiments use one SLM (Qwen3-0.6B) on tool calling. Scores rise on some tasks and fall on others, and the authors themselves say there is no consistent improvement.

Critical strengths

StaTI combines 3 tool-calling benchmarks into one format that can be scored without running the tools. It also provides error breakdowns based on type: wrong function, missing argument, wrong value etc. It is useful for teams working on similar methods.

Critical weaknesses
  1. My read is experiments do not show that the method works -- one SLM, two runs, no error bars, and no comparison other than the unmodified model. 6 of 20 benchmark subtasks (covering more than half examples, 4265 of 7648) are missing from the results table with no explanation).
  2. Each query also costs 8 extra model runs.
  3. Scoring answers by how much likely they are under actual input than under a contrasting distribution is one of the core ideas behind contrastive decoding (Li et al., ACL 2023, https://aclanthology.org/2023.acl-long.687/), which contrasts a strong and a weak model, and context-aware decoding (Shi et al., NAACL 2024, https://aclanthology.org/2024.naacl-short.69/), which contrasts the model with and without the input context. Neither paper is cited or used as a baseline.
Questions for authors
  1. Any reason six subtasks (Live Irrelevance, the four multi-turn tasks, and IBTC) were left out? Could you please report all 20 with confidence intervals and an overall average?
  2. With the same compute budget does SPA beat contrastive decoding (refered above) that uses the alternatives as the contrast with a tuned weight? What about a simple majority vote over 8 sampled answers?
  3. How does the results scale across model sizes, for example a larger model like Qwen3-8B?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.