- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes ARC, a controller that learns when and which repair to apply to an initial answer from a frozen small LLM. All available repairs are applied to the same Anchor offline, providing matched supervision. At deployment, a hierarchical DeBERTa-v3-small router conditioned on the question, answer, and reasoning retains the Anchor or executes one repair.
Critical strengths
- Efficient inference is an important direction.
- The method is clear, straightforward, and easy to follow.
Critical weaknesses
-
The empirical gains are modest, and several simpler variants match or exceed full ARC. The results do not support the design claims.
(a) In Figure 2, the Flat Router stays within ~0.5 points of ARC at matched budgets, and even scores slightly higher with fewer calls.
(b) In Table 2, 4/5 of 7 variants outperform full ARC.
(c) The simpler Family-Aware When-Only reaches 0.6997 at 0.1442 calls, surpassing ARC's 0.6886 at 0.1426 (Figure 4).
-
The presentation of results is confusing and not organized clearly. Key evidence is scattered across the main text and multiple appendices, and different tables and figures draw on different runs.
-
There are internal inconsistencies between results. the ARC vs. Flat Router gap is 2.23 points in Table 2 but shrinks to ~0.5 points in Figure 2. Similarly, ARC outperforms Question-Only by 1.51 points in Table 1, while the gap vanishes in the deterministic evaluation (0.6929 vs. 0.6926).
-
The router must be retrained whenever the base model changes, which limits practical utility. It would substantially strengthen the paper to include transfer experiments or experiments with larger base models.
-
An experiment on iterative/multi-round repair would clarify whether the framework extends beyond the single-step setting.
-
It would be better to clarify that "availability" (line 122) refers to the actions' availability. Introduce Ĝ (line 209) explicitly as the predicted gain. Gives precise definitions of the metrics at their first occurrence.
Questions for authors
See weaknesses