- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper discusses how to adapt a frozen language model by adjusting the sampling. Their goal is to determine when to use constrained decoding and when to use a small expert to adjust logits. They evaluate it on Gorilla, BFCL, and two semantic parsing datasets. They analyze different strategies for constrained decoding and usage of the small expert.
Critical strengths
- The paper covers an interesting and impactful topic (adapting a model when you don't have the ability to adjust the weights).
- The paper evaluates on relevant datasets and provides a lot of analysis.
Critical weaknesses
-
I felt that the writing was confusing and it took a lot of rereading to understand the setup. I will give concrete examples here of what I mean:
-
- Opening sentence, "adapt a frozen, served language model to a private, user-specific inventory, such as a model hub" <- private, user-specific inventory isn't a well-known term. Referring to a user-specific model hub makes me think the user is running their own model hub or something, which is not what is happening here.
-
- "private" in this context is defined in a footnote on page 3, it needs to be much earlier
-
- section 4.2, "where the rules pay" <- this could mean "pays off" (a good thing) or "pays the price" (a bad thing). If I read 3 more lines I see what you're saying but the headings should stand on their own
-
- you need a "teaser figure" that explains the problem setup (including concrete examples) and gives a brief overview of your method.
-
- The usage of "knowledge" in the paper is confusing. E.g. line 058, "the first guarantees legality but cannot add knowledge". Do you simply mean "knowledge of the members of an inventory", or general knowledge in the QA sense?
-
- It would also help if you take time in the intro to precisely define terms like legality (line 59) in bold
-
- line 120: "the default class returns the bare host" <- what's "the default class"? Or did you mean "default case"?
-
Figure 1 isn't especially clear. Is the router or the gate determining whether to go up or down? The gate is right after request (in the same box as router), but it seems the router happens once but the gate happens every token? The flow is not clear.
-
Much of the setup discusses APIs that are closed. Though it's not clear to me how you would use the "Expert" setup, as the logits typically aren't available.
-
The most important evaluations would be on an actual model that can only be accessed by API, given the scenario. It seems teh models are Qwen2.5-7B, which is locally modifiable. Models only accessible through API are often larger and might not demonstrate the same failure modes that are seen in Qwen2.5-7B. Also newer Qwen variants might not be subject to the same failure modes.
Questions for authors
No questions beyond what was mentioned above.
We thank Reviewer 2jfN for the presentation suggestions and API questions; all fixes and new runs are marked in blue.
We replaced "private, user-specific inventory" with user-owned inventory in blue (Abstract, Sections 1 & 3). Footnote 1 (p. 1) now defines it upfront as a finite set of allowed identifiers that belongs to the user and was not used to train the host. In our benchmarks, public hubs serve as reproducible proxies across host-familiarity levels. We also test genuinely unseen user-owned names via ShadowHub (200 invented identifiers, App. B.3) and our 0–100% alias sweep (App. H.1).
Fixed in blue: we moved this definition to Footnote 1 on page 1 (at first mention in Section 1) and removed the old page-3 footnote.
Fixed in blue: we renamed Section 4.2 (p. 6) to "When constrained decoding improves accuracy" so the heading stands on its own.
Fixed: we redesigned Figure 1 as a teaser and moved it from page 3 to page 2. It walks through two concrete requests (a TF Hub lookup and a math question outside every inventory) and shows how the router, mask, and expert handle each.
We mean the former: knowledge of which inventory entry answers the query, not general QA knowledge. Section 1 (p. 2) now says inventory knowledge (which entry answers the query) in blue at that sentence.
Fixed in blue: Section 1 (p. 2) now defines legality (membership in the allowed inventory) in bold at first use.
Yes, the fallback route: we renamed it in blue to default path (requests outside every inventory, answered by the unchanged host; Fig. 1, Sections 3.1 & 3.4).
Fixed in redrawn Figure 1 (p. 2): the router head (green) now sits alone at the entry and decides once per request between the covered and default paths. The per-token gate is removed from that box and noted separately as a step-by-step baseline (Section 3.4, p. 5).
Token-level expert coupling is not used on closed APIs (Fig. 1; Section 4.4, Limitations). Instead, a local router sends only its verdict and legal set to the API's structured-outputs endpoint (Section 4.3 T3; Algorithm 1, p. 22).
We use open-weight hosts (Qwen2.5-7B; Falcon3-10B, Table 35) only where token coupling needs logits. We also evaluate API-only and newer Qwen models directly (Section 4.2, Table 2; Tables 19–20):