acceptodds
Withdrawn from ICLR 2027

SL Algebra: A Control IR for Structured, Grounded, and Auditable LLM Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model inference increasingly combines heterogeneous controls, including structured-output constraints, retrieval, tool execution, and runtime policies. Existing systems typically implement these mechanisms as separate prompts, decoding processors, or callbacks, making their composition difficult to reason about and optimize. We introduce State-Logit (SL) Algebra, a control intermediate representation that separates state operators, which modify auxiliary state or conditioning context, from logit operators, which transform next-token scores. This representation enables formal reasoning about control plans, semantic-preserving rewrites, alternative physical lowerings, and execution traces. We formalize non-anticipatory plans and multiple levels of equivalence, derive normalization laws for exact hard masks, and prove a tokenization-lift theorem for byte-level interface constraints. We further represent grounding as a logical join independent of its physical realization. Experiments across Qwen2.5 and Qwen3 models show that exact constraints achieve 100% structured-output validity, explicit grounding substantially improves QA exact match, and mask fusion provides a mean 1.55× runtime speedup. These results demonstrate that separating control semantics from physical realization enables both formal guarantees and practical optimization for controlled LLM inference.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Models, Intermediate Representation, Structured Generation, Retrieval-Augmented Generation, Runtime Systems

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MpVjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SL Algebra, a control intermediate representation (IR) for structured, grounded, and auditable LLM inference. The proposed framework separates inference-time controls into state operators and logit operators, aiming to support formal reasoning, semantics-preserving transformations, alternative physical implementations, and execution tracing. The authors provide several algebraic properties and evaluate the framework on structured generation and retrieval-augmented question answering tasks using Qwen2.5 and Qwen3 models.

Critical strengths

The paper addresses a relevant problem in LLM inference systems: the composition and management of heterogeneous inference-time controls.

The proposed separation between state operators and logit operators provides a clear conceptual organization.

The paper includes formal analysis, multiple experimental settings, and runtime diagnostics.

Critical weaknesses
  1. Limited methodological novelty and insufficient justification for the proposed IR.

The main technical components, including constrained decoding, retrieval-augmented generation, logit masking, and execution tracing, are already well-established. The primary contribution appears to be reorganizing these existing mechanisms into a unified intermediate representation. However, the paper does not clearly demonstrate what fundamentally new capabilities are enabled by SL Algebra or why existing frameworks cannot achieve similar functionality. The proposed algebraic properties, such as mask fusion, commutativity with finite biases, and idempotence, largely follow directly from the mathematical definitions of these operations. The paper should more clearly distinguish its technical contributions from existing LLM programming and inference frameworks, such as LMQL and SGLang, and constrained-decoding systems.

  1. The experiments do not adequately validate the contribution of SL Algebra itself. This is my primary concern.

The experiments mainly demonstrate that existing inference-time techniques work as expected, rather than showing that the proposed unified representation provides additional benefits. A more meaningful evaluation would compare implementations of identical control mechanisms with and without the proposed IR, demonstrating whether SL Algebra provides additional benefits in execution efficiency, optimization, correctness, or maintainability.

Questions for authors

What concrete functionality or optimization does SL Algebra enable that cannot be readily implemented using existing inference frameworks? Could the authors provide a direct comparison under identical control requirements?

Can the authors demonstrate an end-to-end performance improvement after eliminating repeated constraint compilation, rather than reporting only operator-level mask-fusion speedups?

Reviewer_icJ9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces State-Logit (SL) Algebra, an intermediate representation for controlling LLM inference. It separates state-changing operations, such as retrieval and tool execution, from logit-level operations, such as constrained decoding and token masking. The authors formalize control plans, establish equivalence-preserving transformations for exact hard masks, and introduce logical joins to represent retrieval independently of its implementation.

Experiments on Qwen2.5 and Qwen3 models demonstrate improved structured-output validity through constrained decoding, improved question-answering accuracy through retrieval, and a 1.55× operator-level speedup through mask fusion. The paper also investigates conditional retrieval and execution tracing, although the proposed Guarded Join approach increases latency and reduces accuracy.

Critical strengths

None

Critical weaknesses
  1. The main empirical improvements arise from established techniques such as constrained decoding and RAG. The paper does not clearly demonstrate that SL Algebra enables substantial capabilities or optimizations beyond existing inference frameworks.

  2. The reported 1.55× speedup applies only to the masking operation rather than overall inference. Moreover, Guarded Join increases latency and reduces accuracy. Without stronger end-to-end comparisons against optimized baselines, the practical significance of the proposed IR remains unclear.

Questions for authors
  1. Can the authors provide a concrete example of an optimization or control composition enabled by SL Algebra that cannot be achieved straightforwardly using existing frameworks such as SGLang, LMQL, or vLLM?
  2. The reported 1.55× speedup concerns mask fusion rather than complete inference. What is the end-to-end latency or throughput improvement compared with an optimized baseline that already combines the same constraints?
Reviewer_wscoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

For me the paper is a difficult read. The overall idea seems interesting - seeing the decoding time control strategies of LLMs as state operators and logit operators. State operators are things like retrievals, tool calls, memory, etc. which change the conditioning input to the LLMs. Logit operators change next token probabilities. At every decoding step, state operators are applied, followed by logit operators. After this there will be a sampling to get the actual output. Each control has a check - for example an operator which appends the retrieved documents as part of the context cannot happen while decoding is still being done with the same KV cache. The KV cache is invalidated at this point and hence must be recomputed. After the plan has been validated, it is simplified - if there are multiple masks, they are all combined into one. This is followed by converting the plan to real code (in words). Finally, outputs from the LLM are collated.

Critical strengths
  • Separating the decoding time controls into what the models see as inputs and how the scores are changed is an interesting framing
  • Reproducibility seems to be a strong point of the paper
  • Limitations are clearly mentioned - the authors show that guarded join does not work which I appreciate
Critical weaknesses
  • The different actions on multiple bias masks - commutation, idempotence etc. seems trivial. A negative infinity subsumes all numbers, which is the basis of all the proofs

  • While checking the plan, there seems to be no equivalence mentioned for different lowerings - like early join, memory fusion etc

  • The baseline numbers are very weak in my view - a simple chat templatised form e.g. might increase the 0% to some higher number.

Questions for authors
  • Which equivalence level must a legal lowering of join satisfy?
  • How do the Table 1 results change with the proper chat template and a stop sequence?
  • Can you show a head-to-head against an existing structured-generation system, or one end-to-end case where the IR enables an optimization or guarantee that existing tools cannot provide?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.