- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes the CaRR framework to address the performance degradation of editable DLMs under low inference budgets. While the research context holds some engineering value, the proposed method lacks generalizability, and there are some flaws in the study's motivation and cost claims.
Critical strengths
Practical Significance: Focusing on editable DLMs is cutting-edge and demonstrates certain potential.
Critical weaknesses
- Lack of Generalizability in the SCR Module: The SCR module heavily relies on hard-coded rules, such as AST parsing for code or formatting rules for math. This is essentially a task-specific post-processing engineering trick rather than a fundamental algorithmic innovation, making it difficult to generalize to open-ended natural language tasks.
- Misleading "Compute-Free" Claim: The paper claims that SCR is "compute-free," but entirely ignores the additional memory overhead required to store historical states. Furthermore, the CPU and I/O latency introduced by invoking external verifiers is not accounted for in the total system overhead.
- Questionable Motivation (Forced Justification): In real-world scenarios with extremely limited compute (e.g., a 1/8 budget), the industry typically opts for smaller-parameter models or mature solutions like Speculative Decoding. Artificially restricting the DLM iteration count only to patch it with complex rules fails to prove its indispensability compared to existing optimal acceleration schemes. Moreover, given the severe performance drop under low compute, the rationale for continuing to use such models in these cases remains unclear.
- Concerns on Presentation and Baseline Rigor: The manuscript exhibits heavy traces of AI generation, which raises concerns about the scientific rigor and validity of the experimental conclusions.
Questions for authors
- Could the authors clarify how the SCR module can generalize beyond the specific rule-based tasks (e.g., code and math) evaluated in the paper?
- Could the authors provide a stronger justification for why SCR is preferable to existing low-compute alternatives, such as smaller models or Speculative Decoding, under constrained compute budgets?