- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CONGRATTS, a convergence-guided and resource-aware framework for adaptive test-time scaling of locally hosted LLMs. The method uses a lightweight convergence predictor and runtime resource monitoring to determine when to stop self-consistency sampling or self-refinement. Experiments demonstrate reductions in scaling rounds, latency, and energy consumption across several small language models and hardware platforms.
Critical strengths
- The paper addresses a practically relevant problem in efficient on-device LLM inference.
- The experimental evaluation is relatively comprehensive, covering multiple models, benchmarks, quantization settings, and real hardware.
Critical weaknesses
- Limited methodological novelty. The proposed framework essentially combines a learned convergence-based stopping criterion with a resource-feasibility check. Both components are conceptually straightforward and closely related to existing adaptive stopping and resource-aware inference techniques. The paper does not clearly establish a substantial methodological advance beyond integrating these existing ideas.
- Limited improvement over existing adaptive stopping methods. In self-consistency experiments, CONGRATTS does not consistently outperform existing approaches such as CGES in the accuracy–compute trade-off.
Questions for authors
See weakness
3 comments
The rebuttal did not address my concern about the novelty. The core method is essentially a combination of learned early stopping and resource-threshold checking, both of which are well-established techniques.
We thank the reviewer for their follow-up. We agree that learned stopping and resource-threshold checking are not individually novel. However, our key contribution is a single unified stopping model that generalises across scaling paradigms, a capability that, to our knowledge, has not been demonstrated by any existing method. Our single jointly trained model supports both parallel self-consistency and sequential self-refinement without retraining, and transfers across multiple LLMs, datasets, quantisation settings, and inference backends. We hope this clarifies our intended contribution.
We thank the reviewer for their assessment. We agree that neither a learned stopping criterion nor a resource-feasibility check is novel in itself. Our contribution is the design of a shared, resource-aware continuation framework that is deliberately not specialised to a single scaling paradigm or deployment configuration.
1. Novelty and motivation
Existing adaptive stopping methods are largely specialised to their scaling paradigm. For example, CGES and ASC exploit statistics specific to repeated self-consistency samples, whereas Semantic-Streak and Edit-Saturation operate on changes between successive self-refinement outputs. CONGRATTS instead learns a common notion of output convergence and uses the same jointly trained predictor for both SC and SR. This same learned model is subsequently reused without retraining across three additional LLMs, quantized inference, and a second inference backend.
The resource-feasibility mechanism is an explicit component of our framework, not an inherent property of the existing stopping methods. To make the reviewer-requested hardware comparison controlled, we augmented CGES and semantic-streak with the same CONGRATTS resource-feasibility guard. This isolates the stopping criterion while holding the resource-aware execution mechanism fixed.
The resulting matched comparison is:
These results help clarify our intended contribution. On Mac, CONGRATTS achieves higher SC accuracy than CGES, with query-level analysis showing that additional sampling can recover answers missed by CGES. On Jetson, CGES achieves slightly higher accuracy with fewer rounds, although most CGES-only successes coincide with resource gating that prevents CONGRATTS from refining its initial answer. CONGRATTS also achieves substantially higher SR accuracy on both platforms, using the same learned convergence predictor throughout. We therefore do not claim universal superiority over paradigm-specific stopping methods, particularly as separate hardware executions prevent attributing the differences solely to the stopping criteria.
The objective of CONGRATTS is, instead, to provide a unified transferable controller that supports both parallel and sequential scaling, together with a common resource-feasibility mechanism for live device constraints.
Our design is motivated by the fact that no single scaling paradigm is preferable across applications (as shown in Fig. 1), highlighting the value of supporting different scaling procedures across query and task types. Supporting SC, SR, and potentially other output-sequence scaling paradigms with a common controller reduces the need to develop and maintain separate learned stopping mechanisms for each setting. This is particularly relevant for local inference, where controller complexity and additional model, calibration, and implementation overheads are important.
2. Comparison with existing adaptive stopping methods
We agree that CONGRATTS does not uniformly dominate CGES in the SC accuracy--compute trade-off, and we do not intend to make that claim. In our original evaluation, across the 12 SC model--dataset combinations, CONGRATTS achieves higher mean accuracy than CGES in 9 cases, equal accuracy in one, and lower accuracy in two. Its macro-average accuracy is 62.06% versus 61.67% for CGES, while reducing scaling rounds by 48.2% compared with 62.0% for CGES. The new matched hardware comparison above provides the same picture for SC under identical resource constraints. Importantly, however, the same CONGRATTS predictor also operates on SR, where it achieves higher observed accuracy than Semantic-Streak on both devices and uses fewer rounds on the M1 Max.
Accordingly, we view our contribution as broader than obtaining the best accuracy--compute operating point for every individual SC (or SR) configuration. Unlike existing paradigm-specific stopping methods, CONGRATTS learns a shared convergence model that operates across both dominant scaling paradigms, achieving competitive performance while incorporating live resource feasibility for on-device execution.
We will revise the draft to make this intended contribution clear and to avoid implying universal superiority over specialised stopping methods.