acceptodds
Under review as a conference paper at ICLR 2027

When to Constrain, Guide, or Defer: Adapting Frozen Language Models to Private Inventories

Abstract

In this paper, we explore how to adapt a frozen, served language model to a private, user-specific inventory, such as a model hub, a tool catalog, or a database schema, without modifying its weights. Focusing on the sampling space, we study two complementary mechanisms for inference-time adaptation: constrained decoding and a small expert trained on user-side usage logs. We analyze when each is beneficial, at what granularity it should be applied, and how they should be composed. In our experiments, constrained decoding yields gains only where the base model already possesses partial knowledge of the relevant names (+6 on the primary model where it does, down to where it does not; the boundary holds across the 46 host–catalog pairs we can probe), while the neural expert helps only within its training scope (+21 on its catalog, down to off it). We further find that the signal for deciding when an expert should act is weak in thresholded token-level statistics, only partly recovered by a learned per-token head, and present in a representation of the full request. We propose a prefill-based router that reads the expert's own prefill state to either apply adaptations or default to the base model. The same request-level verdict can activate both mechanisms at once: when both apply, the mask adds its guarantee that no invented name is emitted. We observe that the routed, unmasked coupling adds 16 points of accuracy on its target dataset and returns the base model exactly on unrelated math questions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Model Routing, Constrained Decoding, Domain Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2jfNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper discusses how to adapt a frozen language model by adjusting the sampling. Their goal is to determine when to use constrained decoding and when to use a small expert to adjust logits. They evaluate it on Gorilla, BFCL, and two semantic parsing datasets. They analyze different strategies for constrained decoding and usage of the small expert.

Critical strengths
  • The paper covers an interesting and impactful topic (adapting a model when you don't have the ability to adjust the weights).
  • The paper evaluates on relevant datasets and provides a lot of analysis.
Critical weaknesses
  • I felt that the writing was confusing and it took a lot of rereading to understand the setup. I will give concrete examples here of what I mean:

    • Opening sentence, "adapt a frozen, served language model to a private, user-specific inventory, such as a model hub" <- private, user-specific inventory isn't a well-known term. Referring to a user-specific model hub makes me think the user is running their own model hub or something, which is not what is happening here.
    • "private" in this context is defined in a footnote on page 3, it needs to be much earlier
    • section 4.2, "where the rules pay" <- this could mean "pays off" (a good thing) or "pays the price" (a bad thing). If I read 3 more lines I see what you're saying but the headings should stand on their own
    • you need a "teaser figure" that explains the problem setup (including concrete examples) and gives a brief overview of your method.
    • The usage of "knowledge" in the paper is confusing. E.g. line 058, "the first guarantees legality but cannot add knowledge". Do you simply mean "knowledge of the members of an inventory", or general knowledge in the QA sense?
    • It would also help if you take time in the intro to precisely define terms like legality (line 59) in bold
    • line 120: "the default class returns the bare host" <- what's "the default class"? Or did you mean "default case"?
  • Figure 1 isn't especially clear. Is the router or the gate determining whether to go up or down? The gate is right after request (in the same box as router), but it seems the router happens once but the gate happens every token? The flow is not clear.

  • Much of the setup discusses APIs that are closed. Though it's not clear to me how you would use the "Expert" setup, as the logits typically aren't available.

  • The most important evaluations would be on an actual model that can only be accessed by API, given the scenario. It seems teh models are Qwen2.5-7B, which is locally modifiable. Models only accessible through API are often larger and might not demonstrate the same failure modes that are seen in Qwen2.5-7B. Also newer Qwen variants might not be subject to the same failure modes.

Questions for authors

No questions beyond what was mentioned above.

1 comment

We thank Reviewer 2jfN for the presentation suggestions and API questions; all fixes and new runs are marked in blue.

Opening sentence, "adapt a frozen, served language model to a private, user-specific inventory, such as a model hub" <- private, user-specific inventory isn't a well-known term. Referring to a user-specific model hub makes me think the user is running their own model hub or something, which is not what is happening here.

We replaced "private, user-specific inventory" with user-owned inventory in blue (Abstract, Sections 1 & 3). Footnote 1 (p. 1) now defines it upfront as a finite set of allowed identifiers that belongs to the user and was not used to train the host. In our benchmarks, public hubs serve as reproducible proxies across host-familiarity levels. We also test genuinely unseen user-owned names via ShadowHub (200 invented identifiers, App. B.3) and our 0–100% alias sweep (App. H.1).

"private" in this context is defined in a footnote on page 3, it needs to be much earlier

Fixed in blue: we moved this definition to Footnote 1 on page 1 (at first mention in Section 1) and removed the old page-3 footnote.

section 4.2, "where the rules pay" <- this could mean "pays off" (a good thing) or "pays the price" (a bad thing). If I read 3 more lines I see what you're saying but the headings should stand on their own

Fixed in blue: we renamed Section 4.2 (p. 6) to "When constrained decoding improves accuracy" so the heading stands on its own.

you need a "teaser figure" that explains the problem setup (including concrete examples) and gives a brief overview of your method.

Fixed: we redesigned Figure 1 as a teaser and moved it from page 3 to page 2. It walks through two concrete requests (a TF Hub lookup and a math question outside every inventory) and shows how the router, mask, and expert handle each.

The usage of "knowledge" in the paper is confusing. E.g. line 058, "the first guarantees legality but cannot add knowledge". Do you simply mean "knowledge of the members of an inventory", or general knowledge in the QA sense?

We mean the former: knowledge of which inventory entry answers the query, not general QA knowledge. Section 1 (p. 2) now says inventory knowledge (which entry answers the query) in blue at that sentence.

It would also help if you take time in the intro to precisely define terms like legality (line 59) in bold

Fixed in blue: Section 1 (p. 2) now defines legality (membership in the allowed inventory) in bold at first use.

line 120: "the default class returns the bare host" <- what's "the default class"? Or did you mean "default case"?

Yes, the fallback route: we renamed it in blue to default path (requests outside every inventory, answered by the unchanged host; Fig. 1, Sections 3.1 & 3.4).

Figure 1 isn't especially clear. Is the router or the gate determining whether to go up or down? The gate is right after request (in the same box as router), but it seems the router happens once but the gate happens every token? The flow is not clear.

Fixed in redrawn Figure 1 (p. 2): the router head (green) now sits alone at the entry and decides once per request between the covered and default paths. The per-token gate is removed from that box and noted separately as a step-by-step baseline (Section 3.4, p. 5).

Much of the setup discusses APIs that are closed. Though it's not clear to me how you would use the "Expert" setup, as the logits typically aren't available.

Token-level expert coupling is not used on closed APIs (Fig. 1; Section 4.4, Limitations). Instead, a local router sends only its verdict and legal set to the API's structured-outputs endpoint (Section 4.3 T3; Algorithm 1, p. 22).

The most important evaluations would be on an actual model that can only be accessed by API, given the scenario. It seems teh models are Qwen2.5-7B, which is locally modifiable. Models only accessible through API are often larger and might not demonstrate the same failure modes that are seen in Qwen2.5-7B. Also newer Qwen variants might not be subject to the same failure modes.

We use open-weight hosts (Qwen2.5-7B; Falcon3-10B, Table 35) only where token coupling needs logits. We also evaluate API-only and newer Qwen models directly (Section 4.2, Table 2; Tables 19–20):

  • API-only models (five OpenAI + new Claude Sonnet/Opus 4.6): Larger APIs still miss unprompted catalog names (12.5% and 16.1% free-form on HF for Sonnet/Opus 4.6). Structured outputs add +7.7 and +14.5 on TF Hub (+7.3 and +4.7 on HF), and local routing lifts six of the seven APIs by +2.1 to +8.0 points (within 1.7 of an oracle; Table 27).
  • Newer Qwen (Qwen3-8B): Masking shows the same split: +49.1 on familiar TF Hub (5.5% → 54.6%), versus −2.0 on unfamiliar HF (SR@1=1.3%\mathrm{SR}@1 = 1.3\%).
Reviewer_xbPWOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to help a fixed language model use a private catalog of tools, models, or database names. It combines rules that restrict valid outputs, a small model trained on usage records, and a router that decides when help is needed. The routed combination improves accuracy on its target catalog while preserving the base model's answers on the tested unrelated requests.

Critical strengths
  • The paper clearly separates valid names, correct choices, and deciding when adaptation should apply. Experiments show why each matters.
  • Tests cover multiple catalogs, function calling, SQL, and Cypher, with a broad comparison of models for constrained decoding. Paired confidence intervals help assess the reported differences.
  • The paper reports important failures, including harmful expert guidance outside its catalog and sensitivity to the router's input format.
Critical weaknesses
  1. Preservation outside the target domain is fragile under new traffic. The limitations report 14–99.5% misrouting on unseen general requests under the hub frame, and 17–20% on unseen chat after broader negative training. These results limit the practical reach of the goal of preserving base-model accuracy for any request mixture.
  2. The support-recall result is partly confounded with catalog identity. Figure 2 states that every TF Hub pair has support recall at least 30%, while every HuggingFace and TorchHub pair is at most 10%. The within-TF Hub analysis helps, but more catalogs with overlapping recall ranges are needed to establish a general decision rule.
  3. The added value of expert-based routing is not consistent. A router using generic sentence embeddings nearly matches the trained head on TF Hub and general traffic. The reported benefit mainly concerns HuggingFace recall, so broader evidence is needed to justify the more specialized routing representation.
  4. The full method has narrower deployment evidence than the mask. Closed APIs receive routing and output constraints but cannot use the neural expert. Retrieval also performs better on HuggingFace and TorchHub, so neither the closed-API results nor the broad mask study establishes that the full combination is generally preferable.
Questions for authors

see weakness

1 comment

We thank Reviewer xbPW for the thoughtful review and helpful stress-test suggestions; edits and new controls are marked in blue (see also our General Response).

w1. Preservation outside the target domain is fragile under new traffic. ...

Yes: we narrowed our preservation claim in blue to calibrated traffic (Abstract, Section 3.1 Footnote 2, Sections 4.4–5). Deployments therefore need representative general negatives, as our unseen-traffic tests show:

  • 6-fold domain holdout: Across six general datasets, false activation rises from 1.0% (seen) to 11.1% (held-out), and tighter thresholds trade off catalog recall (App. H.3, Table 38 & Fig. 7).
  • Updated 9-head rates: The cited rates update to 22–95% (hub frame) and 28–34% (unseen chat), with far smaller losses than always-on adaptation (Section 4.4; Tables 28–29).

w2. The support-recall result is partly confounded with catalog identity. Figure 2 states that every TF Hub pair has support recall at least 30%, while every HuggingFace and TorchHub pair is at most 10%. The within-TF Hub analysis helps, but more catalogs with overlapping recall ranges are needed to establish a general decision rule.

Yes: to test familiarity within one catalog, we replaced 0–100% of TF Hub's identifiers with synthetic aliases across 7B, 14B, and 32B hosts (Section 4.2, Fig. 2c; App. H.1, Fig. 6):

  • Flat mask: Gain tracks headroom on 32B (r=0.88,0.89r = 0.88, 0.89) and 14B's permuted control (r=0.85r = 0.85, vs. −0.05-0.05 primary), but not on 7B (0–6 pt headroom; r=−0.10,0.35r = -0.10, 0.35) or across hosts (R2≤0.10R^2 \le 0.10).
  • Anchored mask: Gain does not track identifier headroom (r∈[−0.73,−0.14]r \in [-0.73, -0.14]); it still adds ~26 points at 100% renamed because the host knows the domain anchors.

Accordingly, we narrowed SR@1\mathrm{SR}@1 and headroom in blue to practical diagnostics rather than general decision rules (Abstract, Sections 1, 4.2 & 5).

w3. The added value of expert-based routing is not consistent. A router using generic sentence embeddings nearly matches the trained head on TF Hub and general traffic. The reported benefit mainly concerns HuggingFace recall, so broader evidence is needed to justify the more specialized routing representation.

Yes: the key mechanism is request-level routing, not specialized prefill features. We updated the Abstract, Section 1, and Section 4.3 (T2) in blue to reflect this. We also added multi-seed and unseen-domain representation controls:

  • Fitted traffic (9 heads, n=688n{=}688): Routed TF Hub reaches 65.3% (tuned) vs. 62.1% (MiniLM 10-NN), 56.9% (MiniLM linear), and 56.3% (untuned), with the main edge on HuggingFace recall (Section 4.3 T2; App. F.1).
  • Unseen domains (6-fold holdout): Unframed false activation ties on tuned and untuned states (11.1%), whereas MiniLM lowers false activation (5.7%, 3.3%) at lower catalog recall (65.6%, 49.6% vs. 77.7%; App. H.3, Table 38).

w4. The full method has narrower deployment evidence than the mask. Closed APIs receive routing and output constraints but cannot use the neural expert. Retrieval also performs better on HuggingFace and TorchHub, so neither the closed-API results nor the broad mask study establishes that the full combination is generally preferable.

Yes: we do not propose one universal combination, and instead identify when each mechanism applies (Section 1, Fig. 1), so practitioners can choose what fits their catalog and token budget:

  • Closed APIs: A local router sends its verdict and legal set via structured outputs (Section 4.3 T3). Across seven APIs (n=1,764n{=}1{,}764), this stays within 1.7 points of an oracle and lifts six by +2.1 to +8.0 points (Table 27).
  • Expert vs. prompt retrieval: Without extra prompt tokens, the 0.5B masked expert beats +500-token prompt retrieval on TF Hub (67.4% vs. 58.4%), whereas retrieval wins on HuggingFace (22.7% vs. 17.9%) and TorchHub (18.0% vs. 13.1%), likely because retrieval matches names explicitly while the expert relies on parametric recall (Section 4.2, Table 3). So where the expert has learned the catalog, it also saves the ~500 prompt tokens per request that retrieval adds.
  • Self-hosted options: On routed TF Hub requests, the 0.5B expert alone matches routed coupling (65.1% vs. 65.2%; Table 37), so a deployment whose expert formats reliably can skip the host there (App. H.4). Routed coupling also replicates on a second model family (Falcon3, +23.0; Table 35).
Reviewer_GCA1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies inference-time controlled text generation for language models on tasks involving a user-provided set of models or tools where the frozen LM may lack the relevant domain knowledge. It combines constrained decoding, a small trained expert, and a request-level routing. Experiments and ablations evaluate task accuracy, effects on general-task performance, and settings where these methods work and where they offer little benefit.

Critical strengths
  1. The paper includes extensive experiments and ablations. I appreciate that the authors also explore settings where the methods fail or offer little benefit.
  2. I found the support-recall/headroom analysis interesting. Using the model’s ranking of valid answers to predict when constrained decoding also seems interesting
Critical weaknesses
  1. The paper is extremely hard to read it uses several terms before making clear what they mean exactly (“inventory”, “slots”, “legality”, etc) and I kept having to go to the appendix( + back and forth) to understand what was actually being evaluated. For example, “merged expert + mask” in Table 3 is the small expert running alone, while the headline result in Table 5 (and abstract) uses both models without a mask. I found these changes in configuration quite confusing. The concerns below are based on my understanding of the paper, but in any case I would recommend rewriting to improve the presentation.
  2. The underlying method seems to be a very similar to existing ideas in controlled generation such as [1] (already cited) further it seems that the bulk of the gain come from simply routing to the trained smaller model, so I am not sure how much the proposed method even helps? I also wonder how much of the these methods would help for newer models which are trained to have much better tool calling abilities

[1] Liu, Alisa, et al. "DExperts: Decoding-time controlled text generation with experts and anti-experts." Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Questions for authors
  1. I don’t understand what Proposition 1 tells us about the actual implemented methods or the results. What does it establish here beyond the standard fact that overlapping distributions are hard to distinguish? Does it provide any guarantee/intuition for any of the methods tried out?
  2. Could the authors explain more concretely what “inventory” "value-level idealization" and “legality” mean? Is an inventory simply a set of allowed identifiers, and legality just membership in that set? If so, why doesn't masking ensures legality by construction? In that case it seems that Property 1 seems to be saying that removing invalid choices can fix an (unconstrained) answer only if the model’s best remaining choice (after removing non members) is correct. However, all actual methods implemented are token by token so what, if anything, does Property 1 establish about these methods?
1 comment

We thank Reviewer GCA1 for the constructive feedback; revisions are marked in blue in the updated PDF (see also our General Response)

Weakness 1. The paper is extremely hard to read it uses several terms before making clear ...

Thank you for pointing this out. We revised Sections 1–4.1 in blue so every term (inventory, legality, identifier slots) is defined at first use in the main text (Footnote 1, Section 1, Fig. 1, Section 3.2). Section 4.1 (p. 6) also adds an upfront Configurations paragraph to distinguish Table 3's masked expert (0.5B expert alone under the mask) from Table 5's unmasked routed coupling. Section 1 now also states the three questions the paper answers (when to constrain, guide, or defer).

Weakness 2. The underlying method seems to be a very similar to existing ideas in controlled generation such as [1] (already cited) ...

  • DExperts [1] uses always-on token-level logit addition; our focus is instead when and how to intervene: (1) request-level routing preserves out-of-domain accuracy (Section 4.3, Table 5), (2) masking guarantees legality (Section 3.2, Table 1), and (3) coupling stabilizes formatting across expert seeds (App. H.4).
  • On your second point, in-domain knowledge does come from the 0.5B expert: on TF Hub, top-kk coupling ties the bare expert at 69.6%, 6.0 points above always-on DExperts-style guidance (App. H.2, Table 37). What our pipeline adds is selective routing: always-on coupling loses 32.0 points on GSM8K, while routed coupling keeps the host's accuracy (+0.0; Table 5).
  • On your third point, newer tool-calling models still gain when catalogs are outside the prompt and the host partly knows them: TF Hub masking adds +49.1 on Qwen3-8B and +7.7/+14.5 on Claude Sonnet/Opus 4.6 (Section 4.2, Table 2; Tables 19–20), and routed structured outputs lift Sonnet/Opus 4.6 from 30.0/30.5 to 32.1/33.3 (Section 4.3, Table 27). Conversely, when schemas are in the prompt, frontier models invent ≤0.6% of names and forcing the mask costs ~16 points of abstention accuracy (BFCL, Table 17).

Question 1. ... What does it establish here beyond the standard fact that overlapping distributions are hard to distinguish?

It claims no new math. It applies Le Cam's bound (Section 4.3, p. 9, in blue) to the measured overlap of the per-step KL statistic (binned TV=0.05\mathrm{TV}=0.05, Fig. 3). So any rule that reads this statistic, not just a threshold τ\tau, has error shares summing to at least 0.950.95.

Question 1. ... Does it provide any guarantee/intuition for any of the methods tried out?

It gives intuition, not a guarantee: the bound covers only rules that read one scalar at one step. It explains why the scalar per-step gates we tested fail (Tables 4–5, 14) and motivates moving beyond single-step scalars to hidden states or full-request context: our learned state gate (+15.8 TF Hub, −1.0 HF) and request-level router (+16.2 TF Hub, +0.0 HF/GSM8K; Table 5).

Question 2. Could the authors explain more ...

  • Yes: an inventory is the allowed identifier set (Footnote 1), and legality is membership in it (Section 1). Masking does guarantee legality by construction (Section 3.2; Table 1). Property 1 instead analyzes accuracy (whether the chosen legal entry is correct). We also renamed "value-level idealization" to whole-identifier setting for clarity (Section 3.2, Table 1).

  • Token-by-token decoding commits greedily per prefix rather than over full identifiers. Its accuracy can therefore exceed SR@1\mathrm{SR}@1 (Table 19, App. H.1), so Property 1 is not an exact bound for token-level decoding. Even so, decoding still ends at a legal whole identifier chosen from the host's own probabilities. Property 1's qualitative takeaway thus still holds: the mask can recover only knowledge the host already has, so it helps where the host partly knows the inventory. Across 46 host–catalog pairs, accuracy does not drop at SR@1≥30%\mathrm{SR}@1 \ge 30\% (TF Hub), where the mask improves 16 of the 17 open hosts (+3.9 to +49.1; Section 4.2, Fig. 2a). Drops occur only at SR@1≤10%\mathrm{SR}@1 \le 10\%, where changes stay within ±5.5 points (HF, TorchHub).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.