- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates two function representation strategies for small language models in vehicle function calling: Functional Token (FT) and Schema-in-Prompt (SIP). The authors construct a synthetic benchmark based on the Android Automotive specification, covering Seen Functions, Unseen Functions, and Out-of-Scope Requests. They evaluate both representations across 10 small language models, examining function-calling accuracy, abstention behavior, and inference efficiency. The paper also provides theoretical analyses of the representations' output and information-access limitations and applies split conformal risk control to calibrate abstention decisions. The results show that SIP offers greater flexibility in handling functions not used as training targets and unavailable functions, while FT has substantially lower inference overhead.
Critical strengths
- The paper addresses a practically relevant problem in on-device vehicle function calling, where limited computational resources and changing function availability introduce important constraints. The comparison between FT and SIP provides a useful perspective on the trade-off between function representation flexibility and inference efficiency.
- The experimental evaluation is relatively comprehensive, covering 10 small language models, multiple function-calling scenarios, and five training seeds per main configuration. The benchmark is derived from a documented vehicle API specification, and the authors provide detailed data validation procedures. The additional analyses of abstention behavior, risk calibration, and inference latency further strengthen the empirical scope of the study.
Critical weaknesses
- The benchmark is derived from a single Android Automotive API specification, and its training and evaluation requests are generated through the same synthetic pipeline using one primary language model. The specification-grounded construction and extensive validity checks are valuable, but they do not establish representativeness of naturally occurring driver requests or robustness to independently sourced vehicle interfaces. Although the authors acknowledge these limitations, the absence of external validation limits the extent to which the findings can support broader recommendations for in-vehicle function calling.
- The main FT–SIP comparison changes both the function representation and access to availability information: FT receives only the user request, whereas SIP observes the currently offered schemas. The resulting differences in Out-of-Scope abstention and coverage under risk control therefore do not isolate the effect of representation. Appendix D.5 partially addresses this issue by providing offered-function tokens to FT, but deterministic availability-aware alternatives, such as constrained decoding or post-generation filtering, remain unevaluated. This leaves uncertainty about how much of SIP’s advantage would remain against an FT system equipped with an explicit availability check.
Questions for authors
- The Unseen split excludes functions as training targets but allows their schemas to appear as non-target candidates, and 15 of the 18 Unseen Functions share argument structures with Seen Functions. This evaluates target-held-out generalization, but does not isolate generalization to schemas that are entirely absent from training. Could the authors clarify the scope of the resulting generalization claim and provide a schema-held-out evaluation? If feasible, separately reporting results for familiar versus novel argument structures would help distinguish transfer of existing argument patterns from handling new function definitions.
- Appendix D.5 shows that supplying offered-function tokens to FT narrows the abstention gap, and Appendix C.3 notes that availability-aware decoding removes the information-based obstruction in Theorem 4.2. Would SIP retain its advantage over FT with constrained decoding or a post-generation check that rejects calls to unavailable functions? After recalibrating each resulting system under the same protocol, please compare Out-of-Scope over-triggering, particularly on requests for unavailable Seen Functions, and the Seen right-function rate at the risk budgets used in Figure 3(b).
3 comments
Reply to Reviewer cxw7 (2/2): external validation (W1)
W1. We agree that the benchmark cannot show that its requests match how drivers speak, and we know of no public dataset of naturally spoken in-car control requests.
As a limited external check, we used publicly documented example commands: the voice commands that Google and carmakers publish for cars with Android Automotive. We labelled them before running any model. Then we tested every trained model in the paper: 10 models, 2 representations and 5 seeds, 100 runs in all (details and sources in our reply to Reviewer xat4). A command counts as answered if most of a representation's 50 runs get it right. A call is correct if it uses the right function with the right values; when the command leaves a value open, we only check the function.
- Both representations abstain on all 189 requests our functions cannot serve, including the 20 about the car itself.
- Of the 36 call commands (30 Seen, 6 Unseen), SIP answers 19 Seen and 5 Unseen; FT answers 14 Seen. All of these are among the 27 commands that name the feature they want (22 Seen, 5 Unseen).
- On the 13 commands that state every value, graded exactly, SIP answers 9 of 10 Seen and 3 of 3 Unseen; FT answers 8 of 10.
- Neither answers any of the 9 indirect commands that name no feature, such as "It's hot in here".
- Averaged over seeds, SIP answers more Seen commands than FT in each of the 10 models.
These commands are few and were written to illustrate products. So we do not rank models with them, and they cannot show that the benchmark matches driver speech. The revision reports them in Appendix D.15.
Two limits remain: our requests come from one writer model and one vehicle interface. The independent screen and the hand-checked Verified Slice (Appendix B.7) show that the requests are valid, not that they are representative. The revised conclusion states these limits.
We hope these answers address your concerns, and we are happy to discuss any of them further.
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts in addressing my concerns and revising the paper accordingly. The new results have substantially clarified the key issues, and I am willing to raise my overall score.
Reply to Reviewer cxw7 (1/2): FT with an availability check (W2, Q2) and what Unseen measures (Q1)
Thank you for the detailed review.
We have uploaded a revised PDF, with changes marked in blue. "The revision" below refers to it. We number your points in order: weaknesses W1 and W2, questions Q1 and Q2. W2, Q2 and Q1 are shared with Reviewer SfbM, so the full results of the two new experiments are in our general response (parts 1/3 and 2/3). Our answers are below.
W2 and Q2: does SIP keep its advantage over FT with an availability check? In part. With the mask, FT closes the Seen gap. But at the same risk budget SIP still makes fewer calls when the requested Seen Function is not offered, and only SIP can call Unseen Functions. We agree that the main comparison changes two things at once: the representation, and access to the offered set. We tested both alternatives you name on all 50 trained FT models (10 models, 5 seeds each): constrained decoding (an offered-token mask) and a post-generation filter. We recalibrated each system with the Figure 3(b) protocol. Ranges below are over the 10 models, each a mean over 5 seeds.
General response (1/3) has the table for every risk budget. The revision limits every abstention claim to FT given only the request (abstract, Sections 6.1 and 7), and Theorem 4.2's last clause becomes "every request-only FT rule is of this kind". Sections 6.1 and 6.3 report both variants, and Figure 3 draws the masked one in panel (b) and a new panel (c). The conclusion recommends FT with an availability check when the set of functions is fixed, and SIP with recalibrated abstention when functions are added after training.
Q1: scope of the generalization claim. We agree. An Unseen Function is never a training target, but SIP reads its schema in training as a distractor (a non-target candidate in the offered set). Each Unseen schema appears in the offered set of 36.7 to 38.2% of training examples. The revision states this scope in Sections 3 and 6.2 and gives the counts in Appendix B.4. It describes the main Unseen results as generalization to functions never trained as targets.
Q1: schema-held-out evaluation. We ran it. We retrained all 10 SIP models (5 seeds each) with every Unseen schema removed from the training offered sets, so each Unseen schema is new at test time. Across the 10 models, Unseen accuracy is 15.9 to 76.6% in the main grid and 44.3 to 87.6% after retraining. It rises in every model, by +10.3 to +32.6 points. So the exposure did not inflate the submitted numbers. The cost: when the requested Unseen Function is not offered, the retrained models abstain less often (by 0.5 to 24.1 points). The design and the full results are in general response (2/3), and in Section 6.2 and Appendix D.13 of the revision.
Q1: familiar versus novel argument structures. Twin status does not predict SIP's Unseen accuracy consistently. 15 of the 18 Unseen Functions are twins: their arguments have the same names, types and required set as those of a training function. With functions weighted equally, twins score higher in 5 models and lower in 5 in the main grid. In the schema-held-out grid, non-twins score higher in 8 of 10 models. Only three Unseen Functions are non-twins, so we do not conclude more than this from the split. The revision reports both splits in Appendix D.14 (Table 29).