acceptodds
Under review as a conference paper at ICLR 2027

LatentHarness: Adaptive Latent Context Management for Multi-Agent Systems

Abstract

Latent communication enables LLM agents in multi-agent systems (MAS) to collaborate by exchanging continuous representations. Methods for organizing and delivering these representations fall into two categories: (1) Training-based methods learn message compression or information allocation but require additional training data and parameter optimization. (2) Training-free methods often deliver complete messages or select representations from predefined positions, limiting adaptation to receiver needs. To address this limitation, we introduce LatentHarness, a training-free framework for adaptive latent context management with two components: i) state selection, which adaptively selects hidden-state positions across upstream responses by balancing relevance to the question and receiver role against redundancy; and ii) context construction, which assembles short spans of consecutive hidden states ending at these positions in upstream response and token order. Experiments on four scientific benchmarks with three LLMs show accuracy gains in most settings. On GPQA Diamond, LatentHarness improves Qwen3.5-4B accuracy from 63.77% to 71.74% while reducing textual token volume by 75.57% relative to passing all preceding responses as text. The gains also extend to hierarchical MAS.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Context management

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QNr7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LatentHarness, a training-free approach to selecting which upstream hidden states will be provided to a receiver agent in a latent multi-agent pipeline. Candidates are selected based on sentence boundaries, fixed stride, and response end. Each candidate is evaluated by computing its cosine similarity to a query vector that includes both the question and the receiver's role prompt, as well as the top-k logit-lens token overlap. Then, an MMR-like greedy selection process picks candidates until one of two early stopping criteria is met. For each picked candidate, a span of up to 32 preceding states is created. These spans are then combined, sorted, tagged with role labels, packed into a 256-position budget, and mapped into the input embedding space using the StateBridge closed-form alignment. The experiments were conducted using a four-agent sequential pipeline and a three-expert hierarchical pipeline, with Qwen3.5-4B/9B and Gemma-4-12B on GPQA Diamond, SuperGPQA, BioProBench, and SciCUEval. LatentHarness outperformed both text and StateBridge baselines in most cases while using fewer counted text tokens.

Critical strengths
  • The question of which state to provide to each agent is timely and important, and has been given less attention than the issue of how to represent the state. It is logical to treat this as a selection problem on top of an already established alignment method.
  • This approach is straightforward and does not require any training. The paper provides sufficient detail for implementation. The GPQA Diamond results were consistent across all three models. The ablations are appropriate controls to evaluate.
  • The paper is transparent about the fact that the token savings on SciCUEval are minimal and that the span annotations were not verified by humans.
Critical weaknesses
  • Every chosen state represents a token in a previous response from an upstream source which has been converted into text. An obvious baseline would be to convert the chosen spans back into text and provide those spans as text within the same budget. A random-span baseline using the same budget, span length and sources would also be necessary.

  • StateBridge is the only latent baseline. LatentMAS, KVComm are both training-free and cited, so at least one or two should be compared. A text compression baseline would also be reasonable.

  • The Token metric does not include the delivered hidden states, all upstream generation, the additional forward pass for the query and the LM-head projections for each candidate. In all MAS variants, the Planner, Critic and Refiner still produce complete text responses, therefore total end-to-end cost may not change.

  • Figure 5 appears to have no candidate-position baseline to compare against.

Questions for authors
  • The Gemma BioProBench number in Table 3 (63.85) differs from Table 1 (62.98) because of the subset. This is explained in the text but should be in the caption.
  • Small grammar issues: “let h_c ∈ R^d denotes” and “reducing textual token by 75.57%”.
Reviewer_yDvJOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents LatentHarness, a framework for managing latent context in multi-agent systems without additional model training. It selects upstream hidden-state positions using question and receiver-role relevance together with a redundancy penalty, expands them into ordered spans, and aligns the retained states with the receiver's input embeddings. Experiments use three models on four scientific benchmarks, with sequential and hierarchical workflows, to evaluate accuracy and textual token volume.

Critical strengths

The paper is easy to follow, and the proposed method is straightforward and easy to understand.

Critical weaknesses
  1. Limited Empirical Support for the Motivation. The paper motivates its approach by attributing redundancy, unnecessary communication overhead, and limited receiver adaptation to existing training-free methods. However, the paper does not directly examine whether representative approaches such as CIPHER, LatentMAS, and KVComm exhibit these limitations in the evaluated settings, leaving the empirical basis for this characterization insufficiently established.

  2. Insufficient Experiments. Table 1 includes only two StateBridge variants as latent communication baselines, omitting comparisons with other training-free methods discussed in Related Work, such as CIPHER, LatentMAS, and KVComm. The additional benefit of LatentHarness over these approaches therefore remains unverified.

  3. Incomplete Efficiency Evaluation. The paper explicitly measures textual token volume, excluding upstream generation, transferred hidden states, and additional finalizer inputs. This metric supports the reported reduction in textual tokens but does not establish savings in total computation or communication cost. End-to-end latency, peak GPU memory, and communication-byte measurements, including the overhead of state selection and alignment, are needed to assess the practical efficiency benefit.

Questions for authors

Please see weaknesses above.

Reviewer_qadNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at how to decide which upstream hidden states a receiving agent should see in a latent-communication MAS, without any training. Candidate positions come from sentence boundaries, fixed-stride positions, and response endings. Each candidate is scored by how well it matches a "receiver query," the last hidden state after encoding the question and role prompt. The score combines cosine similarity in hidden space with overlap between the top-k tokens each state predicts through the output head. Positions are then picked greedily with an MMR-style redundancy penalty and a couple of stopping rules. Each pick is expanded into a span of up to 32 states, everything is packed into 256 positions, and the result is mapped into the receiver's embedding space with StateBridge.

Experiments use three models (Qwen3.5-4B/9B, Gemma-4-12B) on four science QA benchmarks, mostly in a planner/critic/refiner/solver pipeline. The method usually beats text passing and StateBridge, and reports large reductions in "textual token volume." Ablations are mostly on GPQA Diamond.

Critical strengths
  1. The comparison against StateBridge is the most convincing part of the paper. LatentHarness is at least as good in all 12 settings, sometimes by a wide margin, while feeding the receiver far fewer states. That is a consistent pattern across models and datasets, and it supports the paper's point that how latent context is organized matters.

  2. I also liked the equal-length replacement experiment. Holding length, alignment, and seed fixed while swapping out the selected spans is the right way to show that the content of the selection matters, not just its size.

Critical weaknesses

My main concern is statistical. As far as I can tell, every number comes from one sampled run at temperature 0.4. GPQA Diamond here has 138 questions, so a single question moves accuracy by 0.72 points, and the standard error of a single run is roughly:

0.7×0.3/138≈3.9 pp\sqrt{0.7 \times 0.3 / 138} \approx 3.9\ \text{pp}

The headline +7.97 is 11 questions. The "w/o relevance" gap and the hierarchical GPQA gain are 2 questions each. On the larger benchmarks, many of the gains are around a point. I don't think the ablation conclusions can be drawn from this. It's also worth noting that on BioProBench the single-agent baseline is within a point of LatentHarness, so some of the improvement over TextMAS seems to come from TextMAS being worse than no collaboration at all.

My second concern is the efficiency claim and what's actually driving the gains. The token metric excludes upstream generation and the transferred states. Upstream generation is up to 8k tokens per agent and the same for every MAS method, so the 75% figure says little about real compute. More importantly, the selected spans could just be decoded and passed as text under the same budget. Without that comparison, I can't tell whether latent communication is helping, or whether this is mainly a good context-selection heuristic that would work just as well in text. I'd also want a StateBridge baseline that sees all preceding agents, since LatentHarness gets that and Figure 3 shows it's worth 6–9 points.

Questions for authors
  1. Can you rerun the main table and ablations with more seeds and report variance or paired tests?
  2. How were the hyperparameters and the per-benchmark strides chosen? Were the 60 GPQA questions not in the eval set used for tuning?
  3. What happens if you pass the same selected spans as plain text?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.