- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies ask-or-proceed decisions in tool calls, where default-filled arguments are valid but may lack evidence. SSPEC recovers unsupported fields from the declared schema and evidence buffer, evaluates exposure at the irreversible dispatch, and spends a bounded interruption budget on parameter groups. Experiments use Ask-before-Plan, ProClare, and Minecraft Dialogue Corpus.
Critical strengths
- The paper targets a real blind spot: default-filled arguments that executors do not represent as choices.
- The separation of evidence recovery, dispatch-time exposure, and budgeted questioning is clear and practically relevant.
Critical weaknesses
- The method assumes the schema fully and correctly enumerates runtime-effective fields, defaults, and effect classes. Middleware-injected defaults, cached/inherited values, and cross-call dependencies are not validated.
- Evaluation is limited and baseline fairness/statistics are unclear. Baselines need the same schema/default/evidence access, and the API-match versus final-pass discrepancy needs explanation.
Questions for authors
- How does SSPEC recover “actually effective” fields when defaults are injected by middleware, values are cached/inherited, or arguments depend on earlier calls?
- Can user authorization or system policy count as valid evidence, and how does the evidence buffer verify its source and freshness?
- Do all baselines receive the same schema/default/evidence access? Please explain the large gap between 92.60 API match and 0.60 final pass for SSPEC.
4 comments
(Continued)
Summarized W2 & Q3: Baseline access, and API match against final pass
A3: We respectfully answer your two questions as follows. (1) In the original comparison the compared policies do not read defaults off the declared schema (Appendix A.12, Lines 1254 to 1256), and following your suggestion, Table R3 gives all five of them the same complete call, schema and evidence buffer, where SSPEC keeps 91.4% recall of the unsupported values against 50.9% for executor self-report. (2) The gap arises because API match (Eq. 10, Line 1137) averages over issued calls, while final pass (Eq. 11, Line 1141) averages over episodes and also requires the goal requirements outside the irreversible dispatch, which no policy asks about and which hold in about half of the episodes (Lines 1169 to 1172). The reviewed pair you cite is withdrawn and replaced, and in the revised Table 1 (Lines 347 to 352) SSPEC reads 91.8% API match and 48.5% final pass, leading every compared policy on both.
Table R3: Comparison under matched information on the default-filled episodes of the travel-planning setting, where every policy receives the same complete call, schema and evidence buffer (the clarification predictor, ask-or-execute and the every-turn rule do not read the call, and their rows equal Table 1 of the revised manuscript).
| Calls issued | Withheld | API match, Eq. 10 (%) | API match, names (%) | Final pass (%) | Unsupported recall (%) | Questions / episode | Exposure (%) | |
|---|---|---|---|---|---|---|---|---|
| No-ask executor | 5997 | 0 | 82.8 ± 0.2 | 96.1 ± 0.4 | 32.2 ± 0.9 | 0.0 ± 0.0 | 0.00 ± 0.00 | 64.2 ± 1.6 |
| Executor self-report | 5997 | 0 | 86.3 ± 0.1 | 96.1 ± 0.4 | 39.3 ± 0.9 | 50.9 ± 2.3 | 0.94 ± 0.05 | 42.9 ± 1.0 |
| Executor self-report, asked at the dispatch | 5997 | 0 | 87.0 ± 0.1 | 96.1 ± 0.4 | 40.7 ± 1.2 | 50.9 ± 2.3 | 0.94 ± 0.05 | 38.6 ± 1.3 |
| Missing-argument trigger | 5997 | 0 | 82.8 ± 0.2 | 96.1 ± 0.4 | 32.2 ± 0.9 | 0.0 ± 0.0 | 0.00 ± 0.00 | 64.2 ± 1.6 |
| Clarification predictor | 5997 | 0 | 84.7 ± 0.2 | 96.1 ± 0.4 | 36.1 ± 1.6 | 30.8 ± 2.1 | 0.48 ± 0.03 | 50.8 ± 1.1 |
| Ask-or-execute | 5997 | 0 | 84.8 ± 0.2 | 96.1 ± 0.4 | 36.8 ± 1.1 | 29.4 ± 3.8 | 0.45 ± 0.06 | 51.3 ± 1.6 |
| Every-turn ask | 5997 | 0 | 86.9 ± 0.5 | 96.1 ± 0.4 | 40.8 ± 1.6 | 100.0 ± 0.0 | 4.00 ± 0.06 | 38.1 ± 1.9 |
| No support test | 5797 | 200 | 92.1 ± 0.3 | 96.0 ± 0.4 | 44.7 ± 1.6 | 100.0 ± 0.0 | 0.85 ± 0.02 | 0.0 ± 0.0 |
| String-match support test | 5988 | 9 | 86.3 ± 0.2 | 96.1 ± 0.4 | 39.1 ± 1.3 | 27.3 ± 1.3 | 0.24 ± 0.02 | 43.6 ± 0.9 |
| Reproduction test of Eq. 3 only | 5979 | 18 | 88.6 ± 0.1 | 96.1 ± 0.4 | 43.4 ± 1.2 | 51.6 ± 2.1 | 0.40 ± 0.02 | 27.6 ± 1.5 |
| Full Eq. 3 (SSPEC) | 5937 | 60 | 91.8 ± 0.3 | 96.1 ± 0.4 | 48.5 ± 1.4 | 91.4 ± 1.3 | 0.61 ± 0.02 | 4.6 ± 0.7 |
Following your suggestions, we have added Table R3 to Section 5.3 as Table 6 (Page 9, Lines 432 to 440) and in full to Appendix A.15 as Table 15 (Page 26, Lines 1383 to 1392).
Best Wishes,
The Authors of Paper 53041
Thank you for the added experiments and clarifications. But SSPEC still relies on the declared schema accurately reflecting runtime fields, defaults, and effect classes, and the core soundness and generality concerns remain unresolved, so I will maintain my original score.
Dear Reviewer ZaUK,
Thank you for your timely response and for your continued engagement with our work.
We are glad that the added experiments and clarifications were helpful. On the reliance on the declared schema, Section 3 states it as a condition, and under inaccurate declarations SSPEC's exposure (21.6% and 17.4%) stays below that of the compared policies with accurate ones (38.1%–64.2%). We will keep these experiments in the final version.
Thank you again for your constructive feedback.
Best Wishes,
The Authors of Paper 53041
Response to Reviewer ZaUK
Dear Reviewer ZaUK:
Thank you for recognizing the task value and the clear method design. We have responded point by point to your concerns (including runtime values, evidence sources, and baseline fairness). In the marked copy appended to the revised PDF (Pages 28 to 56), the revisions made in response to your comments are tagged with @Reviewer ZaUK. We hope our responses resolve your concerns and look forward to further discussion.
Summarized W1 & Q1: Middleware defaults, cached or inherited values, and cross-call dependencies
A1: We respectfully clarify that Eq. 2 reads the value in force on every declared field and tests it against the evidence buffer, and each case thus depends on where its value comes from. First, a cached or inherited value that the executor carries into the call is read as an emitted value (Eq. 2, Lines 191 to 194), while a value the runtime holds without emitting it is outside what Eq. 2 reads. Second, an argument that depends on an earlier call is supported when that call's returned result is committed to the buffer and passes both tests of Eq. 3 for the field (Appendix A.11, Lines 1199 to 1201). Third, a middleware default that differs from the declared one breaks the premise of Eq. 2, which Section 3 (Lines 142 to 145) states as a condition. Following your suggestion, Table R1 shows that a differing middleware default raises exposure from 4.6% to 21.6%, a cross-call dependency raises it to 7.2%, and the inherited-value row equals the full-schema row.
Table R1: SSPEC (b = 1) on the default-filled episodes of the travel-planning setting as the declared interface loses information or departs from runtime behaviour (rates are means with standard deviations over 3 seeds, and Withheld is a total over 1500 episodes).
Following your suggestions, we have added Table R1 to Section 5.3 as Table 7 (Page 9, Lines 443 to 450) and in full to Appendix A.15 as Table 16 (Page 27, Lines 1405 to 1414).
Summarized Q2: Authorization and system policy as evidence, and source and freshness
A2: We respectfully answer your two questions as follows. (1) An authorization the user writes in the dialogue can witness the fields it states, after passing both tests of Eq. 3, while a standalone system-policy message is neither user text nor a tool result and is outside the witness family (Appendix A.11, Lines 1199 to 1201). (2) The source of a span is the turn in which the immutable buffer records it, which the method takes as given without a further provenance check, and freshness is decided by the read, which takes the value of the latest span of the current task. Table R2 shows that the field-conditioned read rejects a stale date that the user later corrects and admits the corrected one, while a type-only read admits both. For a free-form string field, which has no next value, Eq. 3 keeps its reproduction test alone and does not reject a stale span (Appendix A.11, Lines 1238 to 1240).
Table R2: Acceptance by Eq. 3 under a type-only read and the field-conditioned read on constructed booking buffers (Desired is the acceptance the definition of a witness requires).
Following your suggestions, we have added Table R2 to Appendix A.15 as Table 13 (Page 25, Lines 1329 to 1338).