- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents a policy to better assign compute in the agentic coding tasks by exploring four different dimensionalities: the reasoning depth, the horizon, the alternative actions, and the read-only subagent fan-outs, and their combinations. The paper first run some pilot experiments and draw two observations to serve their HARE motivation. The paper then detailed their approach HARE, they select some turns mechanically for more reasoning depth, e.g., edit turns or first/last turns and they decide for the interaction horizon whether to continue.
Critical strengths
- The paper did extensive experiments to show the performance of each configuration.
Critical weaknesses
- The paper only studies the main question under one coding agent, model, benchmark, serving stack, and cost accounting. While I understand scale up the experiment to other models and benchmarks are expensive, but this posts a question and weakness on the generalizability of the approach.
- Section 5.2 reports the negative results that the sparsity is the gain rather than where to select the reasoning depth. This is counter-intuitive, yet the paper does not further explain the reason of it. What about the control experiments of the context length because the sparsity also correlates with the context length, on which some research shows that longer context length (prefill) usually leads to worse results.
In general, I feel this is a paper with very extensive experiments, but lack what can be really drawn from the experiments. There are many learnings that we can derive from the results but the paper fails to deliver them.
Questions for authors
- in terms of the horizon, the context needs to equip with better context management, such as sliding window, llm summarization compaction, or other approaches like [1]. How does HARE's results interacts with these approaches? Does the policy need to be redesigned?
- Observation 2 should be mapped with concrete results, now it reads like without no support. For example, "spend depth at selected turns" because ...
- Figure 3 should be placed in page 3 to ensure readers understand what D,I,W,D are.
- Equation 2 formulation is confusing and the continue_t, D_t, W_t, S_t are left unexplained.
- How I controls the decision points and whether to continue, use another LLM? But it looks like in line 337, page 7, the I is also a constant cap. Then does HARE adds anything on top of the constant capping of I?
- For D, how do we know in advance that it is a end turn? It looks like it is entangled with the I which also contribute to decide whether to continue or to end.
- What is the reducer, another LLM?
- page 5, line 316-317 said that "never in the judge that picks among action proposals" However, Figure 2B shows the opposite that the judge reasoning iperforms the best in the reduction.
- why only two compositions, e.g., I think HARE-Scout, HARE-Width and HARE-Edit can be combined as well. And I don't see any definition of Compositions in Figure 4.
- page 7, line 364-365: "A misplaced reducer survives composition" -> However, lots of compositions perform better than the Judge stack.
- Section 5.3 still presents counter-intuitive results, which lead to further questions. for example, the judge/reducer is bad at width (action) but good at reading explorations reduction. does it mean the judge/reducer model is too weak to handle complex actions?