acceptodds
Under review as a conference paper at ICLR 2027

God Does Not Play Dice: Enforcing Batch Invariance in Agentic Kernel Generation

Abstract

Large language model (LLM) serving systems dynamically batch user requests for GPU execution, so the same request may be processed alongside different requests across runs. If the GPU kernels lack batch invariance (BI), this changing batch context can alter the numerical output for an unchanged input, undermining reproducibility. Existing kernel generation agents focus on correctness and efficiency but overlook BI. We find 19 confirmed BI violations among 83 batch-applicable released artifacts. To address this limitation, we propose BIEvolve, which combines contract-guided static dependency analysis with dynamic validation of bitwise consistency across batch contexts. BIEvolve is designed as a plug-in component for general kernel agents, allowing their original optimization capabilities to be reused while adding BI enforcement. We integrate it as successive acceptance gates in the optimization loop, rejecting structurally risky or dynamically non-invariant candidates before performance benchmarking and returning feedback to the agent. Experiments across three agent frameworks under baseline and BI-aware generation on real-world LLM serving tasks show that BIEvolve improves the BI pass rate from 67.6% to 91.9% while retaining 91.8% of the baseline geometric mean speedup. These results demonstrate that BIEvolve can be incorporated into kernel optimization by agents as an additional reliability objective alongside correctness and speed. Our code is available at https://anonymous.4open.science/r/BIEvolve-50F0/.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Agents, GPU Kernel Optimization, Code Generation, Batch Invariance, Reproducibility

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_57chOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies batch-dependent numerical variation in agent-generated GPU kernels. BIEvolve adds contract-guided static dependency checks and dynamic bitwise comparisons across batch contexts to existing kernel agents. The static contracts identify batch-derived quantities and protect parameters affecting partitioning, reductions, and random streams, while dynamic testing checks the output of a preserved sample under different batch contexts. Failures produce diagnostics that guide subsequent kernel revisions through the original agent's feedback channel. Experiments across three agent frameworks report an increase in BI pass rate among correctness-passing kernels from 67.6% to 91.9%, while retaining 91.8% of the baseline geometric-mean speedup. A separate experiment evaluates the complementary detection capabilities of the static and dynamic components on an independently labeled candidate pool.

Critical strengths
  • A relevant reliability problem supported by concrete evidence. Table 1 identifies 19 confirmed BI violations among 83 batch-applicable released artifacts. This motivates checking a property that ordinary numerical agreement with a reference does not establish.

  • An actionable mechanism integrated with several agent frameworks. The static contracts and dependency-path diagnostics provide specific feedback for revision, while dynamic testing covers realized behavior outside the modeled dependencies. Table 2 compares generation with and without BIEvolve within each of three frameworks under matched optimization-turn budgets, and reports ordinary correctness separately from BI.

  • Useful component evidence and an explicit performance tradeoff. On the shared 200-candidate pool, Table 3 reports detection rates of 83% for static analysis, 55% for dynamic validation, and 92% jointly, with 2% rejection among candidates without a confirmed violation. Table 2 reports an overall BI improvement from 67.6% to 91.9% while retaining 91.8% of the baseline geometric-mean speedup. Together, these results support both detector complementarity and the practical value of searching for efficient invariant kernels.

Critical weaknesses
  • The BI evaluation protocol needs clearer coverage and failure accounting. Section 2.2 defines invariance over changes in batch size, composition, and target position. Appendix C.2 describes preserving sample 0 at position 0 while increasing the batch size, but does not enumerate the batch sizes, companion-sample construction, or context counts for each policy. Position invariance is therefore not directly exercised by the described dynamic protocol, and the relationship between in-loop tests and final-solution evaluation is unclear. Table 2 also reports BI-aware pass rates below 100% despite the acceptance gates. Clarifying whether these failures reflect untested contexts or fallback selection would help interpret the method's coverage. The paper appropriately limits its conclusions to the configured test domain; that domain and the final selection policy should be specified explicitly.
Questions for authors
  • Please list the task-level batch policies, batch sizes, context counts, and companion-sample construction. How do the in-loop and final evaluation contexts differ? Are target-position changes and held-out batch boundaries tested?

  • How are final solutions selected when the BI-aware search does not find a passing candidate within its budget? For final kernels that fail BI evaluation, did they pass the in-loop checks and subsequently fail on other contexts, or were they selected through a fallback? A breakdown of the remaining violations would help clarify the practical limits.

3 comments

Thank you for taking the time to review our manuscript and for the positive evaluation of our work. The feedback has helped us further improve our work.

R1. BI evaluation protocol and remaining failures (Covers Critical Weaknesses, Q1&2)

  • Appendix C.2 does not enumerate the batch sizes, companion-sample construction, or context counts for each policy. Position invariance is therefore not directly exercised by the described dynamic protocol, and the relationship between in-loop tests and final-solution evaluation is unclear. Table 2 also reports BI-aware pass rates below 100% despite the acceptance gates. Clarifying whether these failures reflect untested contexts or fallback selection would help interpret the method's coverage. The paper appropriately limits its conclusions to the configured test domain; that domain and the final selection policy should be specified explicitly.
  • Please list the task-level batch policies, batch sizes, context counts, and companion-sample construction. How do the in-loop and final evaluation contexts differ? Are target-position changes and held-out batch boundaries tested?
  • A breakdown of the remaining violations would help clarify the practical limits.

Response: We thank the reviewer for highlighting the ambiguities. The reported BI pass rates are obtained through an independent, comprehensive audit of the final selected kernels. The audit covers 24 batch sizes (1, 2, 3, 4, 5, 7, 8, 9, 15, 16, 17, 31, 32, 33, 63, 64, 65, 127, 128, 129, 255, 256, 257, 512), three batch compositions (homogeneous, heterogeneous, and heterogeneous batches near tile size boundaries), three target positions (first, middle, and last), and three random seeds, totaling 648 configured test settings per kernel. These settings systematically cover the principal BI dimensions defined in Section 2.2.

In contrast, our search time dynamic BI checks intentionally adopt lightweight, task-dependent policies. Following prior work identifying changes in batch size as a major practical source of inference nondeterminism (He and Thinking Machines Lab, 2025), we design our search time dynamic checks to cover batch size variation. Specifically, general tasks use two batch contexts with sizes {1, 8}, while high risk attention tasks use three contexts with sizes {1, 8, 32}. All checks use one fixed seed and preserve the target sample at position 0. Companion samples are constructed either by generating additional inputs with fixed random seeds or by reusing prepared samples, while keeping the target sample unchanged across batch contexts. To further validate our design choice, we examined 124 BI violations identified by the final audit and found that all inspected violations could plausibly be detected through batch size variation alone. Varying batch size can effectively expose composition-dependent behavior when additional requests alter batch-wide workload statistics. And we did not identify any violations attributable solely to changes in target position.

The search time dynamic checks provide immediate feedback to guide candidate generation, whereas the final audit is conducted separately and its results are not exposed to the agent. Such separation reduces opportunities for reward hacking, analogous to public and hidden tests in coding benchmarks. Moreover, running the complete audit during every search iteration would introduce substantial overhead. We therefore reserve broader batch contexts for final evaluation, including held-out batch boundaries.

Regarding the BI-aware pass rates below 100% in Table 2, these failures arise from additional contexts exercised by the final audit but not detected by the search time checks, rather than from fallback selection. Our inspection of all residual failures identifies batch-dependent tiling as the underlying mechanism, involving different tile parameters across implementations. Changes in tile configuration can alter the partitioning or order of floating point operations, leading to rounding differences and BI violations. However, many tile changes merely redistribute independent computations without affecting numerical results. Consequently, statically prohibiting all batch-dependent tile choices would also reject valid optimizations.

More precise analysis of compiled computations and broader dynamic testing could further eliminate residual failures without unnecessarily restricting performance optimizations. Nevertheless, there is a trade-off between detection performance and search overhead, and our BIEvolve checks are intentionally designed to be lightweight, targeting common BI violations while limiting search overhead. Our results demonstrate promising improvements within this scope. Our future work will investigate adaptive BI checks, where agents automatically refine testing strategies based on accumulated experience while balancing detection performance against overhead. We have added clarifications to appendix D, C.2, and section 4.2.

R2. Selection/fallback behavior (Covers Q2)

  • How are final solutions selected when the BI-aware search does not find a passing candidate within its budget?

Response:

If no candidate passes both the static and dynamic BI checks within the search budget, the run is marked as unsuccessful, and no valid kernel is returned. As with correctness checks, candidates failing either BI check are rejected, and the failure feedback is returned to the agent. In our experiments, all correct BI-aware runs produced at least one candidate that passed both BI checks, so we did not encounter any cases actually requiring fallback. We have added clarifications to the revised manuscript section 4.2.

References

He, H. and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab: Connectionism, 2025.

Thanks for the responses, I maintain my current assessment and consider my concerns addressed.

Reviewer_3naEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies batch invariance: whether a sample's output stays bitwise identical when batch size, composition or position changes. It finds violations in 19 of 83 released batch-applicable kernels. BIEvolve adds static and dynamic checks to kernel-generation agents and returns failures as feedback. Across three frameworks, the reported BI pass rate rises from 67.6% to 91.9%, retaining 91.8% of geometric-mean speedup.

Critical strengths
  1. The paper identifies a practical problem that tolerance-based correctness misses. The 19/83 finding and Figure 2 make the motivation concrete.
  2. The gates fit into three existing frameworks with matched turn budgets. BI pass improves in each, and the combined gate detects more violations than either check alone, at a reported median cost of 10.21 ms per candidate.
  3. The released library is useful. The earlier repository check recorded in this review found that all 37 CPU unit tests passed and that the analyzer rejected the printed Figure 2 example.
Critical weaknesses
  1. The final BI audit is not specified. Table 2 counts kernels with “no confirmed BI violation,” but does not give the batch sizes, positions or compositions used to confirm one. Its percentages imply nine residual violations among correctness-passing BI-aware kernels. Did those kernels pass the gates, or were they fallbacks? If evaluation uses the same contexts as the gates, the improvement could partly reflect filtering against the evaluation test. An independent audit of the saved kernels is needed to interpret the headline result.
  2. Passing the gates does not establish batch invariance. The dynamic check always examines sample 0, using only two or three batch sizes in the released code. It does not independently vary position or composition. The earlier CPU probe found a reduction that passed these checks but changed output at other positions. The static check also depends on names and formatting: renaming the split variables lets the Figure 2 example pass, and the line-based C/C++ analysis misses some multiline dependencies. Table 3 itself shows that the combined gates miss 8% of independently confirmed violations. These limitations call for a narrower claim than “enforcing” BI.
  3. The comparisons leave the benefit of in-loop feedback unclear. The experiments appear to use one run per condition, without variance, a prompt-only BI baseline or post-hoc filtering of vanilla trajectories. The Evolve improvement is small enough that repeated runs matter. Table 3 is an offline detection study, not an ablation of agent search. The release also lacks the task validation adapters, candidate pool, logs and evaluation scripts needed to reproduce Tables 1–3.
Questions for authors
  1. What independent audit produced the Table 1/2 BI labels, and what explains the nine residual violations? Can you re-audit saved kernels with varied target positions, composition at fixed batch size and a wider batch-size range?
  2. How does in-loop feedback compare with simply taking the fastest gate-passing candidate from vanilla trajectories? Please provide paired task results and, if feasible, repeated runs for one framework.
  3. Were the static aliases and rules chosen using any artifacts or runs later included in evaluation? After a static rejection, did agents remove the dependence or just rename/reformat it?
  4. Please state the task count and GM population, and explain the Figure 5 subgroup retentions: they do not recombine to Table 2's overall values under the implied subgroup sizes. Does Table A2 cover only K-Search?
8 comments

Thank you the constructive feedback, which has helped us further improve the clarity and presentation of the manuscript. We provide point-by-point responses below.

R1. The final BI audit and residual violations(Covers W1, Q1)

  • Table 2 counts kernels with “no confirmed BI violation,” but does not give the batch sizes, positions or compositions used to confirm one. Its percentages imply nine residual violations among correctness-passing BI-aware kernels. Did those kernels pass the gates, or were they fallbacks? If evaluation uses the same contexts as the gates, the improvement could partly reflect filtering against the evaluation test. An independent audit of the saved kernels is needed to interpret the headline result.
  • What independent audit produced the Table 1/2 BI labels, and what explains the nine residual violations? Can you re-audit saved kernels with varied target positions, composition at fixed batch size and a wider batch-size range?

Response:

We thank the reviewer for highlighting the ambiguity. The BI pass rates reported in Tables 1 and 2 are determined by an independent audit of the final kernels, rather than the search time BI checks, and it already fulfils the requested re-audit. Specifically, the final audit covers 24 batch sizes (1, 2, 3, 4, 5, 7, 8, 9, 15, 16, 17, 31, 32, 33, 63, 64, 65, 127, 128, 129, 255, 256, 257, and 512), three batch compositions (homogeneous, heterogeneous, and heterogeneous batches near tile-size boundaries), three target positions (first, middle, and last), and three random seeds. This yields 648 combinations, systematically varying batch size, composition, and target position, consistent with the principal BI risk factors identified in prior research (Messina and Scotta, 2026). We additionally emphasize tile size boundaries, where changes in execution strategies may introduce numerical inconsistencies (Gond et al., 2026).

Importantly, this audit is substantially broader than the lightweight search time dynamic checks, which test only a few batch contexts with the target fixed at the first position. We deliberately separate these evaluations for two reasons: (1) withholding the rigorous audit from the agent reduces opportunities for reward hacking, analogous to hidden tests in coding benchmarks; and (2) running the full audit during every optimization iteration would introduce substantial overhead. Thus, the reported BI improvements are independently evaluated rather than measured on the same contexts used for search time filtering.

Regarding the residual violations, these are gate passing final kernels that failed the stricter independent audit, rather than fallbacks. Among all the residual failures we examined, all are caused by batch-dependent tiling. Tile configurations determine how computations are distributed across GPU workers. While many changes merely redistribute independent computations without affecting results, others alter floating point operation order and introduce rounding differences that violate BI.

Simply restricting tile parameters would therefore be overly conservative, rejecting valid optimizations alongside genuine violations. More precise analysis of compiled computations and broader dynamic testing could further eliminate residual failures without unnecessarily restricting performance optimizations. Nevertheless, there is a trade-off between detection performance and search overhead, and our BIEvolve checks are intentionally designed to be lightweight, targeting common BI violations while limiting search overhead. Our results demonstrate promising improvements within this scope. We have revised the manuscript section 4.2, added appendix D to it, to clarify the independent audit and explain the mechanism and design choices behind the residual failures.

R2. Clarification of our method design and coverage (Covers W2, Q3)

  • Passing the gates does not establish batch invariance. The dynamic check always examines sample 0, using only two or three batch sizes in the released code. It does not independently vary position or composition. The earlier CPU probe found a reduction that passed these checks but changed output at other positions. The static check also depends on names and formatting: renaming the split variables lets the Figure 2 example pass, and the line-based C/C++ analysis misses some multiline dependencies. Table 3 itself shows that the combined gates miss 8% of independently confirmed violations. These limitations call for a narrower claim than “enforcing” BI.
  • Were the static aliases and rules chosen using any artifacts or runs later included in evaluation?

Response:

We thank the reviewer for these observations. We respectfully emphasize that these observations largely concern coverage scopes already acknowledged in our manuscript. BIEvolve is explicitly presented as a lightweight, empirical approach rather than a formal BI verifier, both our method and results transparently report and quantify its coverage. For example, Section 6 concludes that BIEvolve removes “most observed violations” rather than guaranteeing invariance. Thus, our use of “enforcing” refers to incorporating BI checks as mandatory acceptance gates during generation. We have clarified the distinction in section 3.3 of the manuscript to avoid ambiguity. While these observations motivate our future improvements, we believe their significance should be assessed in the context of the computational trade-offs and potential empirical gains. We would like to clarify these points below.

The search time dynamic checks deliberately prioritize batch size variation, a major practical source of inference nondeterminism identified by prior work (He and Thinking Machines Lab, 2025). Although these checks do not independently vary target position or composition, changing batch size also introduces different companion requests and can expose composition-dependent BI violations. To further assess our design choice, we examined 124 violations identified by the independent final audit and found that their mechanisms could all plausibly be exposed through batch size variation, and we identified no violations attributable solely to target position changes. The reviewer’s CPU probe illustrates a valid coverage gap, but represents a constructed counterexample rather than a commonly observed failure mode in agent generated kernels. Broader dynamic testing could effectively improve coverage, but at increased search overhead.

Regarding the static analysis module, the protected rules and aliases were designed from prior domain knowledge, independently of any artifacts or agent runs subsequently included in evaluation. As described in Section 3.1, the contracts explicitly enumerate recognized batch symbols and protected parameter aliases based on common kernel implementation conventions. Renaming a variable outside this vocabulary can therefore bypass detection, consistent with our declared scope of contract-guided analysis. While this demonstrates a valid theoretical gap, deliberately renaming protected variables does not necessarily reflect representative failure mode in agent generated kernels. Likewise, Section 3.1 describes C/C++/CUDA analysis as using targeted dependency patterns rather than comprehensive parsing. We acknowledge that this introduces another coverage gap. While future improvements could expand coverage through more comprehensive analysis and broader dynamic checks, our prototype design prioritizes lightweight detection of common BI violations and demonstrates promising empirical improvements. We have added clarifications of our design rationale to appendix C.2 and coverage gaps to section 4.2.

R3 Part 1. Benefits of in-loop feedback and repeated runs (Covers W3, Q2)

  • The comparisons leave the benefit of in-loop feedback unclear. The experiments appear to use one run per condition, without variance, a prompt-only BI baseline or post-hoc filtering of vanilla trajectories. The Evolve improvement is small enough that repeated runs matter.
  • How does in-loop feedback compare with simply taking the fastest gate-passing candidate from vanilla trajectories? Please provide paired task results and, if feasible, repeated runs for one framework.

Response:

We thank the reviewer for the suggestions. To clarify the benefit of in-loop BI feedback and assess the robustness of our results, we conduct three additional experiments: (1) post-hoc filtering of vanilla trajectories, (2) a prompt-only BI baseline, and (3) repeated runs of the Evolve Agent. We have incorporated the supporting results in the revised manuscript, section 4.4 added, appendix E, G added.

(1) Post-hoc filtering

To isolate the benefit of in-loop feedback, we apply the same BIEvolve gates retrospectively to the saved vanilla search trajectories and select the fastest gate-passing candidate for each task. We compare the selected candidates against the final solutions generated with in-loop BIEvolve if the paired tasks are both correct. BI pass rates are determined by the independent audit. We provide aggregate GM speedups for comparison, along with the average native search rank, instead of bulky task-level results.

Table R1. Post-hoc filtering vs. in-loop BIEvolve on paired tasks

FrameworkPost-hoc BI passIn-loop BI passAvg. native search rankGM speedup (post-hoc / in-loop)
K-Search89.2%97.3%3.2432.04 / 35.59
Iterative Agent75.7%89.2%4.0022.16 / 22.66
Evolve Agent89.2%89.2%4.3219.24 / 23.31
Evolve Agent (repeat run)78.4%89.2%3.9717.70 / 23.31

The results in Table R1 show that post-hoc filtering also effectively improves BI pass rates compared with vanilla baseline. However, post-hoc filtering has two important limitations. First, for several tasks, no eligible candidate exists in the entire vanilla trajectory, meaning that the optimization effort produces no candidate satisfying the BI gates. Second, even when eligible candidates exist, they can be substantially slower than the fastest generated implementations, with some ranking as low as around 20th in speed among the saved candidates. These observations highlight the benefit of incorporating BI feedback during optimization rather than filtering only after generation. Once a fast but BI violating candidate becomes the incumbent, subsequent turns may continue optimizing its performance while preserving the violation. In contrast, BIEvolve rejects such candidates immediately and provides diagnostic feedback, redirecting subsequent optimization toward BI alternatives without wasting optimization efforts.

Reviewer_ianfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies batch invariance (BI): under fixed hardware and software, a preserved sample should produce bitwise-identical kernel outputs when the surrounding batch changes. It audits released agent-generated kernels, finding 19 confirmed violations among 83 batch-applicable artifacts, and proposes BIEvolve, a plug-in that adds two acceptance gates to existing kernel agents. A DSL identifies batch-context sources and protected implementation parameters; source-level dependency analysis rejects forbidden data/control dependencies, while a dynamic checker compares a preserved sample across selected batch sizes. Failures become feedback for further optimization. The method is integrated into K-Search, Iterative Agent, and Evolve Agent. With Gemini 3.7 Flash on one H100 NVL setup, reported aggregate BI pass rate increases from 67.6% to 91.9%, while the geometric-mean speedup over task references changes from 28.91× to 26.54× (91.8% retained) and correctness from 94.9% to 95.7%. An offline ablation on 200 independently labeled candidates reports 92% violation detection for the combined checker versus 83% static-only and 55% dynamic-only. The evidence supports a useful practical reliability gate on the configured suite; the breadth of independent BI evaluation and the incremental value of the complete method during generation remain insufficiently established.

Critical strengths

The strongest contribution is identifying a concrete failure mode in the current correctness-and-speed evaluation of agent-generated kernels. The audit finds 19 actual bitwise BI violations among 83 applicable released artifacts, and section 2.2 clearly states that preserving the target sample while varying its context is the property of interest.

Critical weaknesses

The remaining BI failures are not clearly explained. Although candidates must pass both BI checks before benchmarking, some final solutions still fail BI. The paper does not clarify whether these failures arise from additional batch contexts in the final evaluation or from fallback solutions that did not pass the checks. This makes the reported BI pass rates difficult to interpret and leaves uncertainty about how reliably BIEvolve enforces BI.

Questions for authors

What final audit detects the residual BI failures among outputs accepted by the BI-aware pipeline? Please specify its batch sizes, compositions, target positions, seeds and sample counts, its relationship to search-time contexts, and any selection or fallback behavior that can return a candidate that did not pass both gates.

2 comments

Thank you for the helpful suggestions. Our point-by-point responses are provided below.

R1. The remaining BI failures (Covers Critical Weaknesses)

  • Although candidates must pass both BI checks before benchmarking, some final solutions still fail BI. The paper does not clarify whether these failures arise from additional batch contexts in the final evaluation or from fallback solutions that did not pass the checks. This makes the reported BI pass rates difficult to interpret and leaves uncertainty about how reliably BIEvolve enforces BI.

Response: Thank you for pointing out the ambiguity. The reported BI pass rates are based on an independent audit of final solutions, which is more rigorous and includes additional batch contexts beyond those tested during search. We adopt this design for two reasons: (1) withholding the rigorous evaluation environment from the agent reduces opportunities for reward hacking, similar to the separation between public and hidden tests in coding benchmarks; and (2) the full evaluation pipeline is time-consuming, so running it only on selected candidates allows faster iteration during optimization. Among the residual failures we examined, we identified batch-dependent tiling as the primary cause. Tile sizes determine how computations are divided across GPU workers. Some tile changes only redistribute independent computations without affecting their results, while others change the order of floating point operations, violating BI due to rounding differences.

Simply protecting tile parameters would therefore be overly restrictive, rejecting valid optimizations alongside genuine violations. The distinction depends on the actual arithmetic performed, which can be affected by both kernel logic and compiler generated execution. More precise analysis of compiled computations and broader dynamic testing could further eliminate residual failures without unnecessarily restricting performance optimizations. Nevertheless, there is a trade-off between detection performance and search overhead, and our BIEvolve checks are intentionally designed to be lightweight, targeting common BI violations while limiting search overhead. Our results demonstrate promising improvements within this scope. We have revised the section 4.2 of our manuscript to clarify the independent audit and explain the mechanism and design choices behind the residual failures.

R2. Final BI audit (Covers Questions)

  • What final audit detects the residual BI failures among outputs accepted by the BI-aware pipeline? Please specify its batch sizes, compositions, target positions, seeds and sample counts, its relationship to search-time contexts.

Response: The final BI audit covers 24 batch sizes (1, 2, 3, 4, 5, 7, 8, 9, 15, 16, 17, 31, 32, 33, 63, 64, 65, 127, 128, 129, 255, 256, 257, 512), three batch compositions (homogeneous, heterogeneous, and heterogeneous batches near tile size boundaries), three target positions (first, middle, and last), and three seeds, yielding 648 combination settings, systematically covering all principal batch context dimensions identified in prior BI research (Messina and Scotta, 2026). We also test near tile size boundaries, where changes in execution strategies can introduce numerical inconsistencies (Gond et al., 2026). Together with multiple random seeds, these settings provide breadth enough coverage of known BI risk factors.

The final audit is much broader than the search-time dynamic BI checks, which are intentionally lightweight and use only a few batch sizes, with the target fixed at the first position. Also, the final audit test is independent with the search-time dynamic BI checks, their relationship is like public and hidden tests in coding benchmarks. We have added appendix D to clarify the independent audit protocol.

R3. Selection/fallback behavior (Covers Questions)

  • Specify selection or fallback behavior that can return a candidate that did not pass both gates.

Response: There is no fallback in our design. If no candidate passes both gates, the run will be marked unsuccessful, with no valid kernel returned. As with correctness checks, a candidate that fails either the static or dynamic BI check is rejected, and the failure feedback is returned to the agent for revision. Only candidates that pass both gates are eligible for final selection. And we never observed 'no final answer could pass both gates' case. In our experiments, all correct BI-aware runs produced at least one candidate that passed both gates, so no fallback to rejected candidates seemed necessary. We have clarified in section 4.2 of our manuscript.

References

Gond, R., Kamath, A. K., Ramjee, R., and Panwar, A. LLM-42: Enabling Determinism in LLM Inference with Verified Speculation. SOSP, 2026.

Messina, A. and Scotta, S. Introducing Background Temperature to Characterise Hidden Randomness in Large Language Models. TMLR, 2026.

Thank you to the authors for the detailed response and revisions. The clarification of the remaining BI failures, independent final audit, and selection behavior address my main concerns. The additional ablation studies and experimental results further strengthen the evaluation.

I appreciate the authors' efforts in addressing these points. Although some limitations remain in detecting all BI violations, I believe my primary concerns have been sufficiently addressed. I am therefore updating my rating from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.