- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies batch-dependent numerical variation in agent-generated GPU kernels. BIEvolve adds contract-guided static dependency checks and dynamic bitwise comparisons across batch contexts to existing kernel agents. The static contracts identify batch-derived quantities and protect parameters affecting partitioning, reductions, and random streams, while dynamic testing checks the output of a preserved sample under different batch contexts. Failures produce diagnostics that guide subsequent kernel revisions through the original agent's feedback channel. Experiments across three agent frameworks report an increase in BI pass rate among correctness-passing kernels from 67.6% to 91.9%, while retaining 91.8% of the baseline geometric-mean speedup. A separate experiment evaluates the complementary detection capabilities of the static and dynamic components on an independently labeled candidate pool.
Critical strengths
-
A relevant reliability problem supported by concrete evidence. Table 1 identifies 19 confirmed BI violations among 83 batch-applicable released artifacts. This motivates checking a property that ordinary numerical agreement with a reference does not establish.
-
An actionable mechanism integrated with several agent frameworks. The static contracts and dependency-path diagnostics provide specific feedback for revision, while dynamic testing covers realized behavior outside the modeled dependencies. Table 2 compares generation with and without BIEvolve within each of three frameworks under matched optimization-turn budgets, and reports ordinary correctness separately from BI.
-
Useful component evidence and an explicit performance tradeoff. On the shared 200-candidate pool, Table 3 reports detection rates of 83% for static analysis, 55% for dynamic validation, and 92% jointly, with 2% rejection among candidates without a confirmed violation. Table 2 reports an overall BI improvement from 67.6% to 91.9% while retaining 91.8% of the baseline geometric-mean speedup. Together, these results support both detector complementarity and the practical value of searching for efficient invariant kernels.
Critical weaknesses
- The BI evaluation protocol needs clearer coverage and failure accounting. Section 2.2 defines invariance over changes in batch size, composition, and target position. Appendix C.2 describes preserving sample 0 at position 0 while increasing the batch size, but does not enumerate the batch sizes, companion-sample construction, or context counts for each policy. Position invariance is therefore not directly exercised by the described dynamic protocol, and the relationship between in-loop tests and final-solution evaluation is unclear. Table 2 also reports BI-aware pass rates below 100% despite the acceptance gates. Clarifying whether these failures reflect untested contexts or fallback selection would help interpret the method's coverage. The paper appropriately limits its conclusions to the configured test domain; that domain and the final selection policy should be specified explicitly.
Questions for authors
-
Please list the task-level batch policies, batch sizes, context counts, and companion-sample construction. How do the in-loop and final evaluation contexts differ? Are target-position changes and held-out batch boundaries tested?
-
How are final solutions selected when the BI-aware search does not find a passing candidate within its budget? For final kernels that fail BI evaluation, did they pass the in-loop checks and subsequently fail on other contexts, or were they selected through a fallback? A breakdown of the remaining violations would help clarify the practical limits.
3 comments
R2. Selection/fallback behavior (Covers Q2)
- How are final solutions selected when the BI-aware search does not find a passing candidate within its budget?
Response:
If no candidate passes both the static and dynamic BI checks within the search budget, the run is marked as unsuccessful, and no valid kernel is returned. As with correctness checks, candidates failing either BI check are rejected, and the failure feedback is returned to the agent. In our experiments, all correct BI-aware runs produced at least one candidate that passed both BI checks, so we did not encounter any cases actually requiring fallback. We have added clarifications to the revised manuscript section 4.2.
References
He, H. and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab: Connectionism, 2025.
Thanks for the responses, I maintain my current assessment and consider my concerns addressed.
Thank you for taking the time to review our manuscript and for the positive evaluation of our work. The feedback has helped us further improve our work.
R1. BI evaluation protocol and remaining failures (Covers Critical Weaknesses, Q1&2)
Response: We thank the reviewer for highlighting the ambiguities. The reported BI pass rates are obtained through an independent, comprehensive audit of the final selected kernels. The audit covers 24 batch sizes (1, 2, 3, 4, 5, 7, 8, 9, 15, 16, 17, 31, 32, 33, 63, 64, 65, 127, 128, 129, 255, 256, 257, 512), three batch compositions (homogeneous, heterogeneous, and heterogeneous batches near tile size boundaries), three target positions (first, middle, and last), and three random seeds, totaling 648 configured test settings per kernel. These settings systematically cover the principal BI dimensions defined in Section 2.2.
In contrast, our search time dynamic BI checks intentionally adopt lightweight, task-dependent policies. Following prior work identifying changes in batch size as a major practical source of inference nondeterminism (He and Thinking Machines Lab, 2025), we design our search time dynamic checks to cover batch size variation. Specifically, general tasks use two batch contexts with sizes {1, 8}, while high risk attention tasks use three contexts with sizes {1, 8, 32}. All checks use one fixed seed and preserve the target sample at position 0. Companion samples are constructed either by generating additional inputs with fixed random seeds or by reusing prepared samples, while keeping the target sample unchanged across batch contexts. To further validate our design choice, we examined 124 BI violations identified by the final audit and found that all inspected violations could plausibly be detected through batch size variation alone. Varying batch size can effectively expose composition-dependent behavior when additional requests alter batch-wide workload statistics. And we did not identify any violations attributable solely to changes in target position.
The search time dynamic checks provide immediate feedback to guide candidate generation, whereas the final audit is conducted separately and its results are not exposed to the agent. Such separation reduces opportunities for reward hacking, analogous to public and hidden tests in coding benchmarks. Moreover, running the complete audit during every search iteration would introduce substantial overhead. We therefore reserve broader batch contexts for final evaluation, including held-out batch boundaries.
Regarding the BI-aware pass rates below 100% in Table 2, these failures arise from additional contexts exercised by the final audit but not detected by the search time checks, rather than from fallback selection. Our inspection of all residual failures identifies batch-dependent tiling as the underlying mechanism, involving different tile parameters across implementations. Changes in tile configuration can alter the partitioning or order of floating point operations, leading to rounding differences and BI violations. However, many tile changes merely redistribute independent computations without affecting numerical results. Consequently, statically prohibiting all batch-dependent tile choices would also reject valid optimizations.
More precise analysis of compiled computations and broader dynamic testing could further eliminate residual failures without unnecessarily restricting performance optimizations. Nevertheless, there is a trade-off between detection performance and search overhead, and our BIEvolve checks are intentionally designed to be lightweight, targeting common BI violations while limiting search overhead. Our results demonstrate promising improvements within this scope. Our future work will investigate adaptive BI checks, where agents automatically refine testing strategies based on accumulated experience while balancing detection performance against overhead. We have added clarifications to appendix D, C.2, and section 4.2.