acceptodds
Under review as a conference paper at ICLR 2027

From Fixed Keys to Readable Schemas: Small Language Models for Vehicle Agent Function Calls

Abstract

In-vehicle assistants are beginning to act on what drivers say, turning a request such as “turn on cruise control” into a call to one of the vehicle's functions. This Vehicle Function Calling must run on the vehicle's own hardware, which calls for small language models (SLMs), and must abstain when no available function fits, because a wrong call acts on hardware. Since the functions differ across vehicles and change with software updates, the key design question is how to represent them to a small model. A Functional Token (FT) model learns one token per function, whereas a Schema-in-Prompt (SIP) model reads the offered functions' schemas in its prompt. Vehicle Function Calling is a new domain: prior vehicle work uses FT on private data, and no public benchmark compares the two. We answer the question in three connected steps. A formal analysis proves that FT can neither call an Unseen Function, one never used as a training target, nor base its abstention on which functions are offered, whereas SIP can do both. Because abstention needs a guarantee, we calibrate it with split conformal risk control. We release a benchmark from the Android Automotive specification and train 10 SLMs under two billion parameters with matched training for each representation. FT scores zero on Unseen Functions, while SIP reaches 15.9 to 76.6%. When a function trained as a target is not offered, only SIP, which reads the offered set, abstains reliably, though it is much slower to its first call. Across model families, size alone does not order Unseen accuracy. The conformal bound holds on random splits, but not when the requested functions shift. How functions are represented, more than model size, decides what a small in-vehicle model can call.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Functional Tokens, In-Vehicle Assistants, On-Device AI, Schema-in-Prompt, Small Language Models, Vehicle Function Calling

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_cxw7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates two function representation strategies for small language models in vehicle function calling: Functional Token (FT) and Schema-in-Prompt (SIP). The authors construct a synthetic benchmark based on the Android Automotive specification, covering Seen Functions, Unseen Functions, and Out-of-Scope Requests. They evaluate both representations across 10 small language models, examining function-calling accuracy, abstention behavior, and inference efficiency. The paper also provides theoretical analyses of the representations' output and information-access limitations and applies split conformal risk control to calibrate abstention decisions. The results show that SIP offers greater flexibility in handling functions not used as training targets and unavailable functions, while FT has substantially lower inference overhead.

Critical strengths
  1. The paper addresses a practically relevant problem in on-device vehicle function calling, where limited computational resources and changing function availability introduce important constraints. The comparison between FT and SIP provides a useful perspective on the trade-off between function representation flexibility and inference efficiency.
  2. The experimental evaluation is relatively comprehensive, covering 10 small language models, multiple function-calling scenarios, and five training seeds per main configuration. The benchmark is derived from a documented vehicle API specification, and the authors provide detailed data validation procedures. The additional analyses of abstention behavior, risk calibration, and inference latency further strengthen the empirical scope of the study.
Critical weaknesses
  1. The benchmark is derived from a single Android Automotive API specification, and its training and evaluation requests are generated through the same synthetic pipeline using one primary language model. The specification-grounded construction and extensive validity checks are valuable, but they do not establish representativeness of naturally occurring driver requests or robustness to independently sourced vehicle interfaces. Although the authors acknowledge these limitations, the absence of external validation limits the extent to which the findings can support broader recommendations for in-vehicle function calling.
  2. The main FT–SIP comparison changes both the function representation and access to availability information: FT receives only the user request, whereas SIP observes the currently offered schemas. The resulting differences in Out-of-Scope abstention and coverage under risk control therefore do not isolate the effect of representation. Appendix D.5 partially addresses this issue by providing offered-function tokens to FT, but deterministic availability-aware alternatives, such as constrained decoding or post-generation filtering, remain unevaluated. This leaves uncertainty about how much of SIP’s advantage would remain against an FT system equipped with an explicit availability check.
Questions for authors
  1. The Unseen split excludes functions as training targets but allows their schemas to appear as non-target candidates, and 15 of the 18 Unseen Functions share argument structures with Seen Functions. This evaluates target-held-out generalization, but does not isolate generalization to schemas that are entirely absent from training. Could the authors clarify the scope of the resulting generalization claim and provide a schema-held-out evaluation? If feasible, separately reporting results for familiar versus novel argument structures would help distinguish transfer of existing argument patterns from handling new function definitions.
  2. Appendix D.5 shows that supplying offered-function tokens to FT narrows the abstention gap, and Appendix C.3 notes that availability-aware decoding removes the information-based obstruction in Theorem 4.2. Would SIP retain its advantage over FT with constrained decoding or a post-generation check that rejects calls to unavailable functions? After recalibrating each resulting system under the same protocol, please compare Out-of-Scope over-triggering, particularly on requests for unavailable Seen Functions, and the Seen right-function rate at the risk budgets used in Figure 3(b).
3 comments
AuthorsOpenReview ↗

Reply to Reviewer cxw7 (1/2): FT with an availability check (W2, Q2) and what Unseen measures (Q1)

Thank you for the detailed review.

We have uploaded a revised PDF, with changes marked in blue. "The revision" below refers to it. We number your points in order: weaknesses W1 and W2, questions Q1 and Q2. W2, Q2 and Q1 are shared with Reviewer SfbM, so the full results of the two new experiments are in our general response (parts 1/3 and 2/3). Our answers are below.

W2 and Q2: does SIP keep its advantage over FT with an availability check? In part. With the mask, FT closes the Seen gap. But at the same risk budget SIP still makes fewer calls when the requested Seen Function is not offered, and only SIP can call Unseen Functions. We agree that the main comparison changes two things at once: the representation, and access to the offered set. We tested both alternatives you name on all 50 trained FT models (10 models, 5 seeds each): constrained decoding (an offered-token mask) and a post-generation filter. We recalibrated each system with the Figure 3(b) protocol. Ranges below are over the 10 models, each a mean over 5 seeds.

  • Seen right-function rate. The gap closes. At α=0.02\alpha=0.02, FT picks the right Seen function 99.7 to 99.9% of the time with the mask, against 41.3 to 46.1% without it and 98.3 to 100.0% for SIP.
  • Over-triggering on requests for unavailable Seen Functions. At α=0.02\alpha=0.02, masked FT calls a function on 13.4 to 18.9% of them, against 12.0 to 16.4% for SIP and 21.9 to 23.0% for FT as submitted. Masked FT makes more of these calls than SIP in 9 of 10 models at α=0.02\alpha=0.02 and in all 10 at α=0.10\alpha=0.10 (63.2 to 76.4% against 32.2 to 55.4%). The filter makes none. Over all Out-of-Scope requests, the rate is set mostly by α\alpha (at α=0.02\alpha=0.02: 3.9 to 4.0% for masked FT, 3.2 to 3.9% for SIP).
  • Unseen Functions. SIP can call them, and no FT variant can (Theorem 4.1(c)).

General response (1/3) has the table for every risk budget. The revision limits every abstention claim to FT given only the request (abstract, Sections 6.1 and 7), and Theorem 4.2's last clause becomes "every request-only FT rule is of this kind". Sections 6.1 and 6.3 report both variants, and Figure 3 draws the masked one in panel (b) and a new panel (c). The conclusion recommends FT with an availability check when the set of functions is fixed, and SIP with recalibrated abstention when functions are added after training.

Q1: scope of the generalization claim. We agree. An Unseen Function is never a training target, but SIP reads its schema in training as a distractor (a non-target candidate in the offered set). Each Unseen schema appears in the offered set of 36.7 to 38.2% of training examples. The revision states this scope in Sections 3 and 6.2 and gives the counts in Appendix B.4. It describes the main Unseen results as generalization to functions never trained as targets.

Q1: schema-held-out evaluation. We ran it. We retrained all 10 SIP models (5 seeds each) with every Unseen schema removed from the training offered sets, so each Unseen schema is new at test time. Across the 10 models, Unseen accuracy is 15.9 to 76.6% in the main grid and 44.3 to 87.6% after retraining. It rises in every model, by +10.3 to +32.6 points. So the exposure did not inflate the submitted numbers. The cost: when the requested Unseen Function is not offered, the retrained models abstain less often (by 0.5 to 24.1 points). The design and the full results are in general response (2/3), and in Section 6.2 and Appendix D.13 of the revision.

Q1: familiar versus novel argument structures. Twin status does not predict SIP's Unseen accuracy consistently. 15 of the 18 Unseen Functions are twins: their arguments have the same names, types and required set as those of a training function. With functions weighted equally, twins score higher in 5 models and lower in 5 in the main grid. In the schema-held-out grid, non-twins score higher in 8 of 10 models. Only three Unseen Functions are non-twins, so we do not conclude more than this from the split. The revision reports both splits in Appendix D.14 (Table 29).

AuthorsOpenReview ↗

Reply to Reviewer cxw7 (2/2): external validation (W1)

W1. We agree that the benchmark cannot show that its requests match how drivers speak, and we know of no public dataset of naturally spoken in-car control requests.

As a limited external check, we used publicly documented example commands: the voice commands that Google and carmakers publish for cars with Android Automotive. We labelled them before running any model. Then we tested every trained model in the paper: 10 models, 2 representations and 5 seeds, 100 runs in all (details and sources in our reply to Reviewer xat4). A command counts as answered if most of a representation's 50 runs get it right. A call is correct if it uses the right function with the right values; when the command leaves a value open, we only check the function.

  • Both representations abstain on all 189 requests our functions cannot serve, including the 20 about the car itself.
  • Of the 36 call commands (30 Seen, 6 Unseen), SIP answers 19 Seen and 5 Unseen; FT answers 14 Seen. All of these are among the 27 commands that name the feature they want (22 Seen, 5 Unseen).
  • On the 13 commands that state every value, graded exactly, SIP answers 9 of 10 Seen and 3 of 3 Unseen; FT answers 8 of 10.
  • Neither answers any of the 9 indirect commands that name no feature, such as "It's hot in here".
  • Averaged over seeds, SIP answers more Seen commands than FT in each of the 10 models.

These commands are few and were written to illustrate products. So we do not rank models with them, and they cannot show that the benchmark matches driver speech. The revision reports them in Appendix D.15.

Two limits remain: our requests come from one writer model and one vehicle interface. The independent screen and the hand-checked Verified Slice (Appendix B.7) show that the requests are valid, not that they are representative. The revised conclusion states these limits.

We hope these answers address your concerns, and we are happy to discuss any of them further.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts in addressing my concerns and revising the paper accordingly. The new results have substantially clarified the key issues, and I am willing to raise my overall score.

Reviewer_xat4OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper provides a detailed comparison of two different paradigms in Vehicle Agent Function Calls: Functional Token (FT) and Schema-in-Prompt (SIP). It theoretically demonstrates that the FT model cannot invoke functions it has not seen during training, nor can it decide whether to reject a request based on function availability; in contrast, the SIP model can, in theory, invoke functions not present in the training data and can reject requests based on the currently provided schema.

Building on this, this paper constructs the first benchmark for training and evaluating small language models in the vehicle function call scenario and trains and evaluates the two paradigms on a series of small models.

Critical strengths

The paper proposes, for the first time, a public benchmark for Vehicle Function Calling that encompasses both training and testing. This can provide valuable data resources for future developments in this field.

The experiments and analysis in this paper are thorough and comprehensive. Comparisons between FT and SIP were conducted across all ten models, verifying each theoretical claim. The results for the Unseen part were broken down in an interpretable manner. The costs associated with actual deployment were fully taken into account.

Critical weaknesses

My only concern is that, since the benchmarks in this paper are synthetic rather than derived from real-world scenarios, further discussion is needed regarding the extent to which the data constructed in this paper aligns with actual user requests. This may limit the explanatory power of the experiment to some extent.

Questions for authors

Would it be possible for the author to compare the current benchmark data with real-world data, or to manually verify the degree of consistency between the two?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer xat4 (1/2): comparison with publicly documented example commands

Thank you for the positive review and for the concrete question.

We have uploaded a revised PDF, with changes marked in blue. "The revision" below refers to it.

Summary. We know of no public dataset of naturally spoken in-car control requests. Collecting one needs an assistant deployed in real cars and recordings of consenting drivers, which is beyond the resources of this work. The closest check we could make uses publicly documented example commands: the voice commands that Google and carmakers publish for cars with Android Automotive. On these commands, most runs of both representations abstain on every request our functions cannot serve. Of the 22 Seen commands that name the feature they want, SIP answers 19 and FT 14. The clear gap is indirect requests such as "It's hot in here", which name no feature.

What we collected. We gathered 253 distinct commands from Google and carmaker pages and one dealer's list (sources in the next comment). Before running any model, we labelled each one against our Function Surface:

  • Of the 47 commands that ask the car to do something, 36 map to a single call (30 Seen, 6 Unseen).
  • 189 ask for something our functions cannot do, such as navigation, media or messages. The correct answer is to abstain.
  • 28 are not graded, because they need several actions ("Lock my car"), a relative change ("Raise the temperature") or an earlier turn. These are known limits: the paper already places them outside the benchmark's scope (Appendix B.9).

A call counts as correct if it uses the right function with the right values. Some commands leave a value open: they name no seat or level ("Turn the fan speed to 3"), or give a temperature in Fahrenheit while our requests use Celsius. For these, we only check the function.

Results. We ran every trained model in the paper: 10 models, 2 representations and 5 seeds, 100 runs in all. A command counts as answered if most of a representation's 50 runs get it right.

  • Requests our functions cannot serve: both representations abstain on all 189, including the 20 about the car itself. This matches the benchmark's requests for features our functions lack.
  • Seen calls: SIP answers 19 of 30 and FT 14 of 30. Unseen calls: SIP answers 5 of 6; FT cannot call Unseen Functions by design.
  • Commands that name the feature: SIP answers 19 of 22 Seen and all 5 Unseen; FT answers 14 of 22.
  • Indirect commands (9): neither representation answers any of them. Both almost always abstain, treating them like requests the car cannot serve. Our benchmark does not target such requests: each training request is written from a target call and says what it asks for.
  • Without the dealer's list (13 Seen commands, six of them indirect): SIP answers 7 and FT 5.

Appendix D.15 of the revision gives the full breakdown (Table 30), including stricter grading and how often a miss is an abstention rather than a wrong action.

Limits. These commands were published to illustrate products. They are not a sample of how drivers speak, so they cannot show that the benchmark matches driver speech. They point to five kinds of request that the benchmark does not target:

  • Indirect requests, such as "It's hot in here", which name no feature.
  • Requests that need a conversion, such as a temperature in Fahrenheit. Our temperature function takes Celsius.
  • Requests that leave a value open, such as the seat or the level. The benchmark uses no default values (such as the driver's seat), so each request must state every value.
  • Relative changes, such as "Raise the temperature". Our functions change settings, but none reads the car's current state, so the new value cannot be worked out.
  • Requests that need several calls, such as "Lock my car" (one call per door).

The revision names the first three as limits of the benchmark (Appendix B.9). The last two are outside its single-call scope, as the paper already states (Appendix B.9; the conclusion lists several-call requests as future work).

We hope this answers your question, and we are happy to discuss it further.

AuthorsOpenReview ↗

Reply to Reviewer xat4 (2/2): sources of the documented commands

All pages were accessed on 11 November 2026.

Reviewer_SfbMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper aims to address deployment constraints: vehicle function calling runs on vehicle hardware, must abstain when no available function fits, and must support cross-platform porting. The authors compare Functional Tokens (FT) and Schema-in-Prompt (SIP). The former assigns trained functions dedicated output tokens, whereas the latter supplies available function schemas at inference. As a deliverable, they present a synthetic benchmark derived from 7979 Android Automotive controls, including functions withheld as training targets and requests requiring abstention. Overall, I think it is an interesting paper in an emerging community with a useful contribution.

Critical strengths

Overall, the paper is well written, though the authors may consider refining the organization (I will come to that in the next section). As a benchmark, analysis-focused paper, I appreciate the useful, careful documentation of the experimental design and artifacts. Meanwhile, the authors reveal a valuable qualification in their deployment analyses. Specifically, in section 6.3, they test function and request-type shifts and explicitly show that function-identity calibration does not control for wrong arguments. This echoes Theorem 4.3, which states marginal confident-error control under a fixed score and exchangeable calibration or test samples. Together, they provide a useful reference for jointly evaluating representations and abstention, rather than treating syntactic function-call accuracy as a sufficient gauge.

Critical weaknesses

My primary concerns concern the isolation of different components in the experimental design. Specifically, this paper compares output encoding under unequal access to the offered set, which limits the generality and novelty of its structural claims. In their implementation, FT receives only the request, while SIP receives the offered schemas. Theorem 4.1(c) follows directly from excluding held-out functions from FT's decoder, and Theorem 4.2 is an input-indistinguishability argument that applies to any model deprived of the offered set. They are valid statements about these interfaces, but do not establish that token-based representations inherently cannot condition abstention on availability. In my opinion, the practical conclusion would be stronger if it had distinguished the closed, request-only FT baseline from availability-aware token methods throughout.

Beyond the isolation above, the held-out and safety conclusions cover a narrower setting than deployment to newly introduced vehicle functions. In appendix B.4, the authors state that all held-out schemas may appear as training distractors and that 15 of 18 held-out functions share an exact argument signature with a training function. Consequently, the experiment primarily measures selecting a never-targeted function from a familiar schema surface, with extensive argument-structure reuse. It does not directly establish generalization to schemas first introduced after training. Rather than simply emphasizing these settings, I would really appreciate it if the authors could derive practical recommendations or tutorials to address them. For instance, additional results separated by novel argument signature or schema exposure, and a more prominent exact-call risk–coverage analysis, would help identify what transfers to a changing function surface.

Questions for authors

Please refer to the weaknesses above for my primary questions. In addition to these questions, I have a few minor questions related to the actual use cases. For the already-trained offered-set FT probe, what are the seen accuracy, unavailable-Seen-function abstention, prompt length, and latency? Do any existing results apply an explicit offered-token decoding mask? Meanwhile, can we decompose existing SIP predictions into held-out functions with an exact training argument-signature twin and the remaining three? How often did each held-out schema appear as a training distractor?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer SfbM (1/2): isolating the representation, the theorems, and the probe (W1, questions)

Thank you for the careful reading and the concrete questions.

We have uploaded a revised PDF, with changes marked in blue. "The revision" below refers to it. We number your two weaknesses W1 and W2, in order. Both are shared with Reviewer cxw7, so the full results of the two new experiments are in our general response (parts 1/3 and 2/3). Our answers are below.

W1: request-only FT and availability-aware token methods. We agree. FT receives only the request, Theorem 4.2 is a statement about that input, and the submission does not separate request-only FT from token methods that are given the offered set. No submitted result applies an offered-token mask (Appendix C.3 of the submission listed it as not evaluated). We have now tested two such methods on all 50 trained FT models (10 models, 5 seeds each), without retraining. The offered-token mask restricts decoding to the tokens of the offered functions and abstention. The offered-list filter turns a call to a function that is not offered into an abstention. After recalibration with the Figure 3(b) protocol (ranges over the 10 models, means over 5 seeds):

  • With the mask, FT picks the right Seen function 99.7 to 99.9% of the time at α=0.02\alpha=0.02, against 41.3 to 46.1% without it and 98.3 to 100.0% for SIP. So the Seen gap in Figure 3(b) comes from FT's input, as Theorem 4.2 predicts. A token method that is given the offered set can condition abstention on availability.
  • SIP keeps two advantages. It can call Unseen Functions, which no FT variant can (Theorem 4.1(c)). And at the same risk budget, it makes fewer confident calls than masked FT when the requested Seen Function is not offered. Masked FT makes more of these calls in 9 of 10 models at α=0.02\alpha=0.02 and in all 10 at α=0.10\alpha=0.10. The filter makes no such calls.

The revision makes this distinction throughout. Every abstention claim is limited to request-only FT (abstract, Sections 6.1 and 7), and Theorem 4.2's last clause becomes "every request-only FT rule is of this kind". Sections 6.1 and 6.3 and Appendix D.12 report the masked and filtered variants, and Figure 3 draws the masked one.

Practical recommendation. The conclusion now gives one. For a fixed set of functions: FT with an availability check (the mask, or the stricter filter). For functions added after training: SIP, with abstention recalibrated on requests for the new functions, because Section 6.3 shows that a threshold calibrated on other functions does not transfer.

Theorems 4.1(c) and 4.2. We agree that both are short arguments. Their role is to say in advance which failures are structural, and where they come from. The new results bear this out: giving FT the offered set removes the failure of Theorem 4.2, and no variant removes that of Theorem 4.1(c). The revision describes them this way (Section 4.2) and claims no more for them.

The offered-set probe (Appendix D.5). Over its 3 seeds and 4 models, against matched cells on the same seeds (for Qwen3-1.7B, the LoRA cells):

  • Abstention. It abstains on 21.0 to 44.6% of the 92 test requests for an unavailable Seen Function (Appendix D.4), against 2.9 to 5.1% for request-only FT and 81.2 to 93.5% for SIP. Listing the offered tokens in the prompt recovers part of the abstention gap; the mask recovers nearly all of it.
  • Seen accuracy. 95.4 to 98.0%, within 2.1 points of matched request-only FT. It is lower on every seed for both 270M models (by 2.1 and 1.3 points) and not lower on average for the two Qwen models.
  • Prompt length. 36 tokens with the Gemma tokenizer and 34 with Qwen's, against 20 and 18 for request-only FT and 1,804 and 1,689 for SIP (medians over all 3,214 test requests).
  • Latency, on the Device Proxy (a server CPU, not in-car hardware). The probe's prompt processing takes 1.59 to 1.72 times as long as request-only FT's, in all four models. For the three models under 1B, its time to the first complete answer (a call or an abstention) is 108 to 203 ms, against 81 to 135 ms for request-only FT. On the same three models, SIP takes 2.3 to 5.8 s, 19 to 29 times the probe. For Qwen3-1.7B the probe takes 452 to 642 ms per seed. We give no ratio for that model, because two of the three 4-bit exports of its request-only FT cell abstain on most of the timed requests.
  • How latency was measured. The three systems were measured in one session, on each model's Q4_K_M export. The server is llama.cpp on a Xeon Platinum 8580, pinned to four cores, with its host-memory prompt cache off. Each figure is the median over 3 seeds of each seed's median over repeated runs, on 20 timed Seen requests. This is the protocol of every Device Proxy figure in the revision (Appendix D.10), which reports the probe in Table 25.
AuthorsOpenReview ↗

Reply to Reviewer SfbM (2/2): what Unseen measures (W2, questions)

W2: schema exposure. In SIP training, each Unseen schema appears as a distractor in the offered set of 36.7 to 38.2% of training examples (2,420 to 2,519 examples). Also, 92 training examples (1.4%) ask for one of 14 Unseen Functions whose schema is not offered, with abstention as the target. No training example has an Unseen Function as its target. The revision gives these figures in Appendix B.4 and states the scope in Sections 3 and 6.2: an Unseen Function is never a training target, but its schema is read in training. Exposure as a distractor is nearly the same for every function, so it does not explain why their accuracies range from 8.5% to 97.5% (mean over the 10 models).

W2: schemas first introduced after training. We agree that the submitted experiment does not show this, so we ran it. We retrained all 10 SIP models (5 seeds each) with every Unseen schema removed from the training offered sets, so each Unseen schema is new at test time. Across the 10 models, Unseen accuracy is 15.9 to 76.6% in the main grid and 44.3 to 87.6% after retraining. It rises in every model, by +10.3 to +32.6 points. So seeing a schema in training only as a distractor makes the model less likely to call it, and this exposure did not inflate the submitted Unseen numbers. The cost: when the requested Unseen Function is not offered, the retrained models abstain less often (by 0.5 to 24.1 points). The design and the full results are in general response (2/3), and in Section 6.2 and Appendix D.13 of the revision. The recommendation for functions added after training (part 1/2) rests on this experiment.

W2: novel argument signatures. Twin status does not predict SIP's Unseen accuracy consistently. 15 of the 18 Unseen Functions are twins: their arguments have the same names, types and required set as those of a training function; allowed values may differ (the revision words Appendix B.4 this way). With functions weighted equally, twins score higher in 5 models and lower in 5 in the main grid. In the schema-held-out grid, non-twins score higher in 8 of 10 models. Only three Unseen Functions are non-twins, so we do not conclude more than this from the split. The revision reports both splits as a table (Appendix D.14, Table 29), and general response (2/3) summarizes them.

W2: exact-call risk-coverage. Section 6.3 already reports two exact-call figures. Calibrating on the function-level loss, the exact-call error reaches 11.1% for the worst SIP cell at α=0.05\alpha=0.05. Calibrating on the exact-call loss itself keeps it at most 5.0% at the same α\alpha. The revision adds the Seen exact-call rate next to each right-function rate under risk control (Section 6.3). At α=0.02\alpha=0.02 it is 98.3 to 99.3% for FT with the mask, against 97.0 to 99.2% for SIP.

We hope these answers address your concerns, and we are happy to discuss any of them further.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/3): FT with an availability check (concern shared by Reviewers cxw7 and SfbM)

We thank the three reviewers for their careful reviews.

We have uploaded a revised PDF, with changes marked in blue. This general response has three parts:

  • (1/3) A new experiment that gives FT the offered set. It answers a concern shared by Reviewers cxw7 and SfbM.
  • (2/3) A new experiment that removes the Unseen schemas from training. It answers a second concern shared by both.
  • (3/3) The list of changes in the revised PDF.

Each reviewer's thread answers every point of that review, and points here for the details of the two experiments. We number each reviewer's weaknesses (W1, W2) and questions (Q1, Q2) in the order of the review.

The concern (cxw7 W2 and Q2, SfbM W1). The main comparison changes two things at once: the representation, and access to the offered set. We agree.

The experiment. We gave FT the offered set in two ways, on all 50 trained FT models (10 models, 5 seeds each), without retraining:

  • Offered-token mask (constrained decoding): FT may only output the token of an offered function, or abstain.
  • Offered-list filter (a post-generation filter): if FT calls a function that is not offered, the call becomes an abstention.

We then recalibrated every system with the Figure 3(b) protocol, each on the same 500 Seen and Out-of-Scope requests.

What we found.

  1. FT's Seen gap in Figure 3(b) comes from its input, as Theorem 4.2 predicts. At α=0.02\alpha=0.02, FT picks the right Seen function 99.7 to 99.9% of the time with the mask and 99.4 to 99.6% with the filter, against 41.3 to 46.1% without either. At every α\alpha, masked FT is at most 0.1 points below SIP (at most 0.5 points for exact calls), at no prompt cost.
  2. SIP keeps one advantage over every FT variant: it can call Unseen Functions (accuracy 15.9 to 76.6%). Every FT variant scores 0.0 there (Theorem 4.1(c)).
  3. Against masked FT, SIP keeps a second advantage. At the same risk budget, it makes fewer confident calls when the requested Seen Function is not offered. Masked FT makes more of these calls than SIP in 9 of 10 models at α=0.02\alpha=0.02 (masked FT minus SIP: -2.7 to +4.9 points) and in all 10 at α=0.10\alpha=0.10 (+9.4 to +33.2).
  4. The filter makes no such calls at any α\alpha, because it abstains whenever FT's choice is not offered. The mask, like SIP in almost every case, chooses among the offered functions, so it is the like-for-like comparison.
  5. Before calibration (each model's own greedy output), the mask abstains on requests for a Seen Function that is not offered more often than SIP (97.2 to 99.6% against 84.6 to 95.0%). The reason is that it removes the requested function's token. Calibration then lowers its threshold toward the risk budget, so more of these requests get a call than under SIP.
  6. Before calibration, Out-of-Scope accuracy is 81.1 to 83.3% for FT given only the request, 99.0 to 99.9% with the mask, 99.5 to 100.0% with the filter, and 96.6 to 98.6% for SIP.
  7. Over all Out-of-Scope requests, the commit rate after calibration is set mostly by α\alpha. At α=0.02\alpha=0.02 it is 3.9 to 4.0% for masked FT and 3.2 to 3.9% for SIP.

Evidence. Percent, mean over 5 seeds, range over the 10 models. "Seen RF" is the rate of picking the right Seen function. "Commit" is the rate of calling a function when the requested Seen Function is not offered (over-triggering).

α\alphaSeen RF: FTSeen RF: FT + maskSeen RF: SIPCommit: FTCommit: FT + maskCommit: SIPModels where FT + mask commits more than SIP
0.0123.2 to 26.799.3 to 99.993.0 to 99.910.7 to 11.56.2 to 9.45.8 to 8.39 of 10
0.0241.3 to 46.199.7 to 99.998.3 to 100.021.9 to 23.013.4 to 18.912.0 to 16.49 of 10
0.0576.4 to 83.099.8 to 100.098.7 to 100.055.3 to 57.533.4 to 45.123.2 to 35.79 of 10
0.1098.7 to 99.499.9 to 100.098.8 to 100.097.0 to 99.363.2 to 76.432.2 to 55.410 of 10
0.2099.4 to 99.699.9 to 100.098.8 to 100.0100.088.4 to 96.945.1 to 76.910 of 10

In the revision. Sections 6.1 and 6.3 report both variants. Figure 3 adds masked FT to panel (b), and a new panel (c) shows the commit rate across α\alpha. Appendix D.12 gives the results per model (Tables 26 and 27).

AuthorsOpenReview ↗

General response (2/3): training without the Unseen schemas, and argument shapes (concern shared by Reviewers cxw7 and SfbM)

The concern (cxw7 Q1, SfbM W2). An Unseen Function is never a training target. But SIP does read its schema in training, as a distractor (a non-target candidate in the offered set). So the submitted results do not show generalization to schemas first seen at test time. We agree.

Exposure in the submitted training set.

  • Each Unseen schema appears as a distractor in the offered set of 36.7 to 38.2% of training examples (2,420 to 2,519 examples). Every training offered set includes 2 to 18 Unseen schemas.
  • 92 training examples (1.4%) ask for one of 14 Unseen Functions whose schema is not offered, with abstention as the target.
  • No training example has an Unseen Function as its target.

The experiment. We retrained all 10 SIP models (5 seeds each) on a modified training set. In it, every Unseen schema in an offered set is replaced by a Seen schema, and each offered set keeps its size. Requests, targets and the test set are unchanged, so at test time every Unseen schema is new. The 92 abstention examples keep their requests, which describe the function in words but show no schema. FT needs no retraining, because it never sees the offered set.

What we found. Removing the Unseen schemas from training raises Unseen accuracy in every model. So the exposure did not inflate the submitted numbers.

Range across the 10 models (%):

Main SIPSchema-held-out SIPChange per model (points)
Unseen accuracy, requests pooled as in the paper15.9 to 76.644.3 to 87.6+10.3 to +32.6
Share of Unseen requests answered with a call30.8 to 80.079.3 to 93.5
  • With the eight held-out property groups weighted equally, as in Appendix D.6, Unseen accuracy rises by +7.1 to +21.8 points, separable from zero in every model.
  • The cost is on requests for an Unseen Function that is not offered (39 test requests). Abstention on them falls by 0.5 to 24.1 points. Most of this fall comes from calls to a sibling function of the same group (another mirror or lumbar axis, for example), made on up to 21.0% of these requests.
  • Seen accuracy changes by -1.0 to 0.0 points, small enough that it may be retraining noise. Out-of-Scope accuracy, which includes the 39 requests above, changes by -2.0 to +0.1.

So seeing a schema in training only as a distractor makes the model less likely to call it. Models that never saw the Unseen schemas call them more often and mostly correctly. They also pick a sibling more often when the requested function is not offered.

Familiar versus new argument structures. Twin status does not predict SIP's Unseen accuracy consistently, with or without exposure. A twin is an Unseen Function whose arguments have the same names, types and required set as those of some training function; allowed values may differ. 15 of the 18 Unseen Functions are twins.

  • Main grid: with functions weighted equally, twins score higher in 5 models and lower in 5. Pooling requests instead, twins score lower in 8 of 10 models, but 65% of twin requests come from five movement functions (mirrors and lumbar supports). The lowest and highest per-function accuracies (8.5% and 97.5%, mean over the 10 models) both belong to twins.
  • Schema-held-out grid: with functions weighted equally, non-twins score higher in 8 of 10 models (the exceptions are the two 270M models).

Only three Unseen Functions are non-twins, so we do not conclude more than this from the split.

In the revision. Section 3 and Appendix B.4 state what Unseen means and give the exposure counts. Section 6.2 reports the retrained grid, with results per model in Appendix D.13 (Table 28). Appendix D.14 gives both splits by argument structure (Table 29).

AuthorsOpenReview ↗

General response (3/3): list of changes in the revised PDF

Changed text is marked in blue. The Device Proxy values in Tables 18 to 20 and Figure 8 are not coloured: we re-measured all of them under a stricter protocol, described under Device Proxy measurements below. The main text is 10 pages. Every section, table and figure of the submission keeps its number; new appendices and tables come after them.

Scope of the claims about FT (general response 1/3)

  • Abstract, Sections 1, 6.1 and 7, Appendix C.3: every abstention claim is limited to FT given only the request ("request-only FT"). The abstract and Section 1 also state the mask result briefly.
  • Section 4: Section 4.1 defines a request-only rule, and Theorem 4.2's last clause becomes "every request-only FT rule is of this kind". Section 4.2 now says what Theorems 4.1(c) and 4.2 are for: they say in advance which failures are structural.

New results

  • FT with the offered-token mask and the offered-list filter: Sections 6.1 and 6.3, Appendix D.12 (Tables 26 and 27). Figure 3 draws masked FT in panel (b) and in a new panel (c). Section 6.1 cites prior work on constrained decoding where it introduces the mask (Geng et al., 2023, the one new reference).
  • The Seen exact-call rate next to each right-function rate under risk control: Section 6.3.
  • What Unseen means, how often each Unseen schema appears in training, and the definition of argument-shape twins: Section 3 and Appendix B.4.
  • SIP retrained without the Unseen schemas (general response 2/3): Section 6.2, Appendix D.13 (Table 28), and one sentence in Section 1 and in Appendix C.4.
  • Unseen accuracy by argument structure: Appendix D.14 (Table 29).
  • Cost of the offered-set probe, with its prompt length and Device Proxy latency: Appendix D.5 (Table 25).
  • Publicly documented example commands, with their sources: Appendix D.15 (Tables 30 and 31).

Conclusion and limits

  • Section 7 recommends FT with an availability check when the set of functions is fixed, and SIP with recalibrated abstention when functions are added after training.
  • Section 7 states the limits: one writer model and one vehicle interface. Appendix B.9 names three kinds of request that the benchmark does not target: indirect requests, requests that need a conversion, and requests that leave a value open.
  • The Ethics statement names the hand-checked labels of the documented commands.
  • The Reproducibility statement says that the scripts behind the new results (Tables 25 to 31, Figure 3(c)) will join the released code. The supplementary material is unchanged.

Device Proxy measurements (Section 6.4, Appendices D.10 and D.11)

  • Every Device Proxy figure is re-measured under a stricter protocol. The server's host-memory prompt cache is off, the weights are read into the server's own memory, and the server is pinned to four cores. Each figure is a median over repeated runs.
  • SIP's time to first call is now 27.1 to 43.6 times FT's (19.7 to 39.5 in the submission). Peak memory is now measured without the prompt cache. No conclusion of Section 6.4 changes.
  • Two 4-bit accuracy values change by 0.1 point (Section 6.4 and Table 21).

Wording

  • Section 1 now says that the 0.6B model "scores higher than" every larger general-purpose model near 1B (the submission said "matches or exceeds"). Section 6.2 gives the paired differences.
  • Appendices D.1, D.5 and D.10 state the role of the LoRA cells, the probe's seeds, and how each Device Proxy figure is aggregated.
  • Two sentences are removed. Appendix B.4 no longer reads Unseen accuracy as mostly binding a new name to a familiar argument shape; the twin analysis does not support that reading. Appendix D.10 no longer reads the generation rates as limited by memory traffic, because the host's memory bandwidth was not measured.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.