- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines how undocumented backend behavior affects LLM tool simulators, whether plausibility judges detect the resulting errors, and which information channels improve simulation. The controlled testbed contains three author-built SQLite servers with 38 hidden rules, including 31 that never trigger in the demonstrations. Source-derived specifications substantially improve first-encounter fidelity over documentation alone and outperform the tested probing configurations on the main student. Generic source summarization performs similarly to the specialized compilation prompt. Downstream experiments show repairs on two selected failure cases and regressions from incorrect probing-derived rules. A public-MCP pilot reproduces documentation gaps and simulation-induced agent failures, but does not reproduce the specification-based repair result.
Critical strengths
- The study separates several questions that are often conflated. The protocol distinguishes teacher-forced response fidelity, reference-free error detection, real-replay task success, agent-declared completion, and simulator-side grading. Section 6 gives a consequential example: a source-conditioned simulator can return the wrong exact capped value yet still provide enough information for the agent to recover. Conversely, a plausible success response can suppress recovery and produce false completion. This is more informative than reporting a single simulator accuracy score.
- The controls support an information-channel interpretation and expose important negative results. Generic summaries, raw-source prompting, a second teacher family, a second student, and artifact redraws help distinguish access to source from the particular SIBYL prompt. The manuscript also reports that the preregistered downstream crossover fails and that source-derived cards do not reliably transfer to public servers. These results sharpen the contribution: readable source is useful in the controlled setting, but compiling it into a reliable simulator specification is not automatically solved.
Critical weaknesses
- The practical repair claim has a substantial external-validity gap. The synthetic servers contain roughly one thousand tokens of source each, with explicitly enumerable author-designed rules. In the public-server pilot, generic summaries improve exact match only modestly, SIBYL lowers outcome agreement, and most raw-source gains concern wording or layout rather than semantic hidden behavior. More importantly, Appendix C reports 113/126 real-replay successes with docs+source versus 116/126 with documentation alone. Thus, the public experiment establishes that simulation errors matter, but does not establish that the proposed information channel repairs them. This does not invalidate the controlled findings, but it weakens the concluding recommendation to provide source as a practical remedy. A targeted analysis of why source fails on the public semantic cases would be more valuable than simply expanding aggregate exact-match evaluation.
- The evidence supports a local channel comparison more strongly than a general hierarchy of channels or compilation methods. Section 5 deliberately gives source arms complete implementation information while probing must discover rules, and teacher effort is not normalized. That is legitimate for comparing available access regimes, but cannot establish greater resource efficiency or superiority over stronger exploration. The source-over-probing contrast also weakens on strict-29, with the second student, and under server-blocked resampling. Similarly, the wide interval for generic-summary versus SIBYL in Table 3 does not establish equivalence between compilation methods. The appropriate conclusion is that no reliable advantage of the specialized prompt was detected, while source access produced large gains over documentation in this inventory.
Questions for authors
- Scope clarification: Would you explicitly separate the three supported conclusions—documentation gaps exist, reference-free checks can miss them, and source helps on the controlled testbed—from the unestablished claim that source-derived specifications improve real-server agent outcomes?
- Mechanism clarification: For public-server semantic errors that remain under raw-source prompting, how much failure comes from missing dependency behavior, incorrect state computation, or failure to apply behavior present in the supplied source? This distinction would materially affect the practical recommendation.