acceptodds
Under review as a conference paper at ICLR 2027

ToolTrials: Agents Learn Opaque Tool Contracts by Experiment

Abstract

An agent can read a tool's schema and still choose the wrong action: labels such as standard, compat, and strict often name distinct operational contracts whose semantics are visible only through execution. Existing approaches rewrite documentation or accumulate task trajectories, but the examples they observe need not distinguish competing behaviors. We introduce ToolTrials, which treats a new tool as an active system-identification problem. Given a finite executable candidate library containing the target and a legal query pool, it maintains candidate contracts, selects the input-policy trial that maximally partitions them, executes the trial, and compiles the identified mapping into a reusable behavioral card. Across 48 opaque implementations, eight behavior families, 288 held-out tasks, and 10,272 planned agent decisions, actively selected receipts reach 84.0-93.1% success versus 53.1-54.5% for passive observations. Compiling those receipts into cards reaches 98.3-99.0%, the oracle range. A matched control applies the identical compiler to model-designed probes and reaches 97.6-100.0%, localizing the downstream gain to verified contract compilation while ToolTrials supplies deterministic, auditable probe selection. Three independent passive draws preserve 44.3-50.7-point card gains. A prospectively frozen non-randomized study identifies 18/18 installed Python, Node, Ruby, and GNU provider/mode contracts and predicts 90/90 held-out outputs; two random trials identify 52.4%. On a separately blinded natural-mode study, opaque case identifiers eliminate answer cues: cards solve 120/120 decisions across three model families, versus 105/120 from schema alone and 107/120 from fixed receipts. Deterministic transfer additionally verifies 480 GNU-wrapper actions and 756 two-tool plans. Two stored trials detect all 480 tested policy-mapping changes, while five-fold repetition preserves 98.5-98.6% identification under 10% output noise. ToolTrials turns tool exploration from incidental trial-and-error into a small, planned, transferable experiment whose evidence and conclusion are executable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language agents, tool use, active system identification, tool contracts, experimentation, behavioral cards, agent reliability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mSnEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper picks a couple of test calls that reveal which setting does what (setting here is about standard or strict for tools where the exact effects arent documented). It then gives the agent a short note with the answer, and with that note agents almost always choose the right setting.

The puzzle being solved is very small. There are three settings, three known behaviours, and the right answer is guaranteed to be on the list, so two well-chosen test calls always solve it. The authors’ own comparison shows that letting the AI design its own test calls works just as well. So the gain comes from handing the agent the answer, not from the paper’s method of choosing the calls

Several headline numbers follow automatically from the setup rather than measuring anything. One table also doesn’t match another, in a study where the authors admit some agents could read the answer files.

Critical strengths

The examples drawn from real software, such as different languages rounding numbers differently, show the problem exists in practice. The authors also ran a comparison that undercuts their own main claim, and they report it directly.

Critical weaknesses

Addressed in the summary.

Questions for authors
  1. Why do the per-family results for one model average 43% when the main table reports 32%?
  2. Is there any situation where your method of choosing test calls beats simply letting the AI design its own?
Reviewer_1GjmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

ToolTrials studies how agents can identify tool behavior that is not fully specified by an interface schema. Given a finite executable candidate library containing the target contract and a legal query pool, it selects trials that maximize information gain, eliminates candidates inconsistent with observed outputs, and compiles the result into a reusable behavioral card containing the inferred mapping and execution receipts. For three known behaviors assigned bijectively to three policy labels, the paper establishes a two-trial identification bound when a separating input exists. The main evaluation covers eight behavior families, 48 mappings, and 288 held-out action-selection tasks. Two actively selected receipts yield 84.0–93.1% success versus 53.1–54.5% for passive examples; compiled cards reach 98.3–99.0%. Applying the same compiler to model-designed probes achieves 97.6–100.0%, showing that both sources of identifying evidence support high downstream accuracy. Additional studies identify 18 installed provider/mode contracts and predict 90 held-out outputs. On 40 natural-mode tasks across three models, cards produce 120 correct decisions, versus 105 from schema alone. GNU-wrapper and two-tool composition experiments verify deterministic execution, and further experiments examine output noise, mapping changes, and selected forms of candidate misspecification. The cards can be shared across compatible agents and replayed to check previously observed tool behavior.

Critical strengths
  • The controlled experiment is comprehensive.
Critical weaknesses
  • Substantial prior specification limits practical applicability: ToolTrials requires an executable candidate library containing the true contract and a predefined legal query pool, while most controlled instances reduce to identifying permutations of three already specified behaviors (§2.1; Table 1). Constructing and validating those candidates is left upstream, so the reported query efficiency does not capture a potentially substantial part of learning an unfamiliar tool. The main evaluation therefore leaves unclear when this preparation is practical and how much benefit remains for larger or less completely specified contract spaces (§§5 and 7).

  • The added value of active selection over strong self-exploration remains unclear: Model-designed probes already identify all 48 controlled mappings within the two-call budget, and applying the same compiler achieves 100.0%/97.6% success compared with ToolTrials’ 98.3%/99.0% (Table 3). These results establish the value of verified contract compilation more clearly than an empirical advantage for the proposed selector. The natural-contract agent study does not include this matched model-probe-plus-compiler baseline, leaving the selector’s incremental benefit on authentic tool contracts unresolved (§4.4).

Questions for authors
  • How should the method balance identifying a complete contract for future reuse against learning only the behavioral distinctions needed to solve the current task?

  • On larger natural-tool candidate libraries, how does ToolTrials compare with model-designed probes using the identical compiler across matched query budgets, when measuring identification accuracy, downstream task success, and the total cost of candidate preparation and execution?

Reviewer_aKBCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper targets tools whose legal mode labels (e.g., standard/compat/strict) hide different behaviors. ToolTrials assumes a finite library of executable candidate behaviors that contains the installed one, plus a pool of legal queries. It enumerates all label-to-behavior bijections, greedily executes the query whose predicted outputs split the surviving hypotheses with maximum entropy, and stores the identified mapping with the observed input–output pairs ("receipts") as a "behavioral card" for LLM agents. With three behaviors, a bijection, and one separating input, two calls are necessary and sufficient. Evaluation uses author-built suites. In the main one, two LLMs solve 288 tasks over 48 synthetic implementations under eight conditions, including a control that compiles model-designed probes. The others are deterministic checks on GNU wrappers and two-tool plans, 18 installed runtime behaviors with a 40-task blinded agent study, and simulated noise, drift, and misspecification tests.

Critical strengths
  1. Transparent, well-controlled evaluation.

    • Missing outputs count as failures against planned denominators (Table 3).
    • Inference treats the families as units, with exact sign tests (p4, L213–215).
    • Appendix F (p12, L620–628) invalidates a Gemini run whose schema sessions "directly read the raw benchmark containing target mappings and answers".
    • The authors report the Self-explore+compiler control (Table 3) even though it narrows their own claim.

    This makes the results trustworthy and yields the paper's clearest finding: deterministic filtering recovers 96/96 mappings, compared with 86 for an LLM rewrite (p6, L309–311).

  2. A real failure mode and an auditable artifact. The merge example (p1, L40–44) shows type-correct labels that hide different null semantics. Two ordinary examples identify only 1 of 18 installed runtime behaviors (5.6%, p7, L369–370). Receipts make cards cheap to revalidate (Sec. 4.5) and make misspecification fail loudly ("detects all 576/576 cases", p8, L403). Tool registries could adopt this mitigation regardless of the selection rule.

Critical weaknesses
  1. Identification is close to trivial under the stated assumptions, and the paper's own control shows that active selection does not drive the gains.

    Results that hold by construction. The method assumes "a finite executable candidate library containing the target and a legal query pool" (p1, L16–17). For the two-call result it also assumes a bijection and "one input separating all behaviors" (Table 1, p3). Then "Because the mapping is bijective, B − 1 trials reveal the final assignment by elimination" (p3, L153–154). The following therefore hold by construction:

    • 100% identification (Table 4).
    • GNU/composition 100% ("deterministic executor verification, not agent planning", p2, L90–91).
    • ToolTrials ≈ Oracle, because the card is the correct mapping plus receipts.
    • 480/480 drift detection (p8, L397–399).

    What the control shows. Model-designed probes compiled the same way reach 100.0% and 97.6%. ToolTrials differs from them by −1.7 and +1.4 points (exact p = .25 and .75; p5, L264–267), and those probes already identify all 48 mappings (p5, L259–261).

    Prior work. What remains is "deterministic, auditable probe selection" (p1, L24–25) by greedy partition entropy. This is essentially greedy version-space splitting [1] and the distinguishing-input queries of oracle-guided synthesis [2]. Information-gain query selection with LLMs also exists [3]. Related Work gives this lineage one sentence (p9, L453–455). Yet the title, and the claim that ToolTrials "uniquely fixes probe selection" (p2, L95–96), present selection as the advance. The evidence supports the card, not the named contribution.

    Suggestion: Reframe the paper around the receipt-backed card, and position the selector against [1–3]. Test selection against an LLM given the same library, in settings where selection can matter: large or non-bijective hypothesis sets without a separating input, costly calls, or noisy outputs, where greedy splitting is fragile [4].

  2. The evaluation never tests the hard regime, and key baselines are uninformative by design.

    • Schema baseline. Each family includes all six bijections, so any fixed reading of the labels is right one time in three. The claim that "schemas alone yield only 31.6–31.9% exact action success" (p1, L49–50) reflects this design. Without randomized aliases (p5, L231), schema alone solves 105/120 blinded decisions (p7, L373–374), and GLM's card-vs-schema interval is [0.0, 17.5] (p7, L377).
    • Passive and random baselines. The controlled query pool, and how "Passive gives two random observations" (p4, L196–197), are never specified, yet they set these baselines. In the natural study the fixed baseline is, by construction, the ordinary inputs at indices 0–1, beside curated edge cases at 2–5 (p11, L576–578). So the 100% vs 5.6% result (Table 6) mostly reflects pool curation.
    • Documentation-learning baselines. Explore+rewrite is "a matched DRAFT/Play2Prompt-style loop under the same two-call budget" (p4, L201–202), whereas DRAFT [5] and Play2Prompt [6] iterate over multiple rounds. ToolObserver and OpaqueToolsBench [7] are not used, although the authors cite that work (p1, L52–53; p9, L448–449).
    • Candidate library. Building a library that contains the target is left as "upstream work" (p9, L432–433).

    As a result, no experiment shows selection helping when a tool's behaviors are unknown, incomplete, or numerous, which is the practically hard case.

    Suggestion: Evaluate on OpaqueToolsBench [7] against ToolObserver and against DRAFT/Play2Prompt at their native budgets. Add LLM-proposed candidate libraries that may miss the target. Report the pool's composition and how passive examples are sampled.

Questions for authors
  1. Table 10's GLM Schema row (p12, L611) averages 43.4%, but Table 3 reports 31.6%. With 36 tasks per family (p4, L191–193) these should agree, and cells such as 72.2% are hard to reconcile with randomized labels. DeepSeek ToolTrials/Schedule = 97.1 is also unattainable with 36 tasks. Which runs and denominators does Table 10 use? Does it include the five excluded tool-using GLM schema batches (p4, L205–207; p12, L624–626)? A clean reconciliation would raise my Soundness score. If Table 3 is the one in error, the "31.6–31.9%" headline changes.
  2. How many inputs per family does the controlled pool contain, what fraction separates all three behaviors, and how are passive observations drawn? If most inputs are non-separating by construction, then the 29.5–39.9-point passive gap (p5, L253–254) and the random curve (Table 4) measure pool design rather than selection, and I would discount them further.
  3. Can you show ToolTrials' selection beating an LLM given the same candidate library? Candidate settings: non-bijective or larger hypothesis spaces without a separating input, LLM-proposed libraries that may miss the target, or OpaqueToolsBench [7] against ToolObserver and DRAFT/Play2Prompt [5, 6] at their native budgets. Such evidence would raise my Informativeness score and Rating.

References

[1] Dasgupta. "Analysis of a greedy active learning strategy." NIPS, 2004.

[2] Jha et al. "Oracle-Guided Component-Based Program Synthesis." ICSE, 2010.

[3] Choudhury et al. "BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Design." ICLR, 2026.

[4] Golovin et al. "Near-Optimal Bayesian Active Learning with Noisy Observations." NIPS, 2010.

[5] Qu et al. "From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions." ICLR, 2025.

[6] Fang et al. "PLAY2PROMPT: Zero-shot Tool Instruction Optimization for LLM Agents via Tool Play." Findings of ACL, 2025.

[7] Hallinan et al. "OpaqueToolsBench: Learning Nuances of Tool Behavior Through Interaction." arXiv:2602.15197, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.