- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper compares three ways to improve an LLM agent under a fixed compute budget: training model weights, rewriting the agent's harness (prompts and skills), or combining both. The experiments show that weight training generally improves performance as the budget increases, while harness rewriting brings little benefit. Combining both helps in some settings, but the improvement does not generalize across tasks.
Critical strengths
The paper studies a practical and important question: where should we spend the compute budget to improve an agent?
The experiments provide a fair comparison by accounting for compute costs, and include different model sizes and environments.
The analysis provides useful insights into why harness rewriting may fail, including the performance drop caused by changing the harness during training.
Critical weaknesses
The paper only tests one harness optimization method, making it difficult to generalize the negative results to other methods.
The experiments mainly focus on relatively simple tasks, so it is unclear whether the conclusions hold for more complex, long-horizon agents.
Questions for authors
Is the limitation in harness optimization or in the model's ability to use the harness? The experiments show that the current rewriting method brings little improvement, but this does not necessarily mean that better harnesses cannot improve the agent. How can we distinguish these two possibilities?
The actual compute usage ranges from 52% to 122% of the allocated budget, with systematic differences across methods. Could these differences affect the conclusions, especially when comparing joint optimization with weight training?