- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper frames omission of future components in multi-agent LLM workflows as a counterfactual credit-assignment problem. The proposed LW2S method collects full-workflow and single-component skip traces, trains action-specific skip-safety predictors, calibrates acceptance thresholds, and combines them with domain-specific guards and sequential fall-through decisions. The method is evaluated on mathematical reasoning (MATH and GSM8K), multiple-choice question answering (MMLU), and code generation (MBPP), using Qwen2.5-7B and Mistral-7B. The authors report reduced component-token cost while matching or improving aggregate accuracy in the evaluated settings. The paper further studies model-family transfer, a larger model diagnostic, and a debate-refine workflow topology.
Critical strengths
- I consider inference-cost reduction in multi-agent LLM workflows an important problem, particularly when workflows execute several components even when some may be redundant.
- I appreciate the use of matched full-workflow and component-skipping traces to construct supervision for skip decisions.
- I appreciate the inclusion of mathematical reasoning, general knowledge, and code generation benchmarks, as well as multiple model families.
Critical weaknesses
- I am concerned that the safety objective measures non-degradation rather than correctness. Since , a skip action is considered safe even if it preserves an incorrect answer. I would like the authors to report how often skipping preserves existing errors, repairs errors, or damages correct answers, clarifying whether LW2S improves task correctness or merely preserves baseline reward.
Questions for authors
- How much of the method performance improvement comes from executable-test guards rather than learned skip selection?
- How robust is the method when task distributions, component implementations, or workflow topologies differ?
- What are the total data-collection and training costs, and how many future workflow executions are required for the method to relieve these costs?