- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies when inference-time scaffolding helps LLM agents, distinguishing workflow decomposition from guidance. It derives a finite-attempt granularity bound, an exact retention-gap identity, and a stopping rule for partial harnessing under homogeneous scaffold reliability and convex autonomous tail risk. Experiments show non-monotonic effects of adding structure, suggesting that scaffolding should match agent capabilities and available evidence.
Critical strengths
-
The workflow/guidance distinction clarifies harness failures. The granularity analysis accounts for tolerance, cumulative action scale, and budget.
-
The fixed-prefix retention-gap identity and stopping rule are clear and correct under their stated assumptions. The paper also explains when the unimodality conclusion can fail.
-
Experiments across ten Terminal-Bench tasks favor shallower scaffolding, with 38% versus 30% success at sizes 2 and 10. Calibration, alternative judges, and human agreement strengthen the hallucination evidence.
Critical weaknesses
-
The stated assumptions do not justify two general design claims. Increasing guidance strength need not preserve the sign of the retention gap: even exponential weighting can favor recoverable paths at low strength and harmful paths at high strength if the latter contain a rare, high-scoring trajectory. Likewise, lower pointwise autonomous tail risk does not guarantee earlier optimal stopping without an ordering assumption on marginal risk reductions. These issues limit the general interpretations while leaving the exact identity and conditional stopping theorem intact.
-
The real-task evidence is difficult to assess. The main sweep uses one run per task/depth, and the expanded success comparison lacks explicit trial counts and uncertainty. Reported aligned hallucination rates differ substantially, while model labels and filtering comparisons need clarification. These differences may reflect separate configurations, which should be specified before combining the results as evidence for one mechanism.
Questions for authors
- What additional assumptions justify sign-preserving amplification and earlier stopping as autonomous capability improves? If these assumptions do not hold, will the general claims be narrowed?