acceptodds
Withdrawn from ICLR 2027

What Do Agents Miss When They Decide? Exploring Epistemic Gaps in LLM-based Multi-Agent Systems through Sibling Probing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM-based multi-agent systems distribute decision making across agents, such as planning and execution. Recent research has shown that failures can arise when a decision lacks necessary information support despite correct execution, a phenomenon known as an epistemic gap. Existing approaches mainly use retrospective feedback from past interactions to calibrate subsequent decisions. However, such feedback offers limited insight into which decision contains an epistemic gap, as decisions are made top-down with obtained information, while later information emerges bottom-up and outcomes entangle multiple decisions. We formulate decision-level epistemic exploration in MAS to acquire information that resolves such gaps. Two challenges arise: epistemic gaps can accumulate across hierarchical decision levels, making it unclear which decision level to explore; explored trajectories entangle information across decision levels, making decision-specific information requirements difficult to extract. We design HIVE, which starts from an observed failure and probes alternatives in a bottom-up order to mitigate the effects of epistemic gap accumulation, while constructing explored trajectories to disentangle role-specific information requirements through trajectory comparisons. Based on these comparisons, HIVE identifies why current knowledge fails to support the decision and updates role-specific knowledge online within the current task. Across six benchmarks, HIVE improves task success by 4.46% over the compared baselines while reducing average trajectory length by 23.67%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-based Multi-Agent Systems, Epistemic Gaps, Sibling Probing, Epistemic Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8EtSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HIVE to address failures caused by insufficient decision-relevant information in LLM-based MAS. HIVE performs hierarchical sibling probing over parameters, tools, and subgoals, and updates role-specific knowledge online through sibling trajectory comparisons. Experiments report consistent improvements in mean performance.

Critical strengths
  • The paper provides a concrete approach to decision-level information acquisition for failure recovery. It expands probing from parameters to tools and subgoals, then incorporates the outcomes into Planner and Executor knowledge. This enables execution-grounded comparisons beyond a verbal explanation of failure.

  • The evaluation coverage and reported results (value itself - see weaknesses 3) are encouraging. HIVE achieves the highest mean accuracy across six benchmarks and additional LLM backbones, while the ablations in Table 2 support the usefulness of hierarchical sibling probing and online knowledge updating.

Critical weaknesses
  • The claims about identifying and disentangling information requirements exceed the demonstrated evidence. HIVE expands the scope of changes from parameters to tools and subgoals, rather than descending through successor states as in conventional DFS. However, successful hierarchical exploration alone does not establish the correct identification of missing information.

  • The experiments demonstrate improvements in mean performance but do not sufficiently establish the added value over generic exploration and reflection or statistical significance. Without budget-matched comparisons against DFS-like exploration with reflection memory, simple retries, and alternative probing orders, or independent validation of information requirements, the benefits of additional exploration remain difficult to separate from those of the proposed comparison and knowledge updating procedures.

  • The paper lacks statistical reporting. Providing the standard deviation for each reported metric alongside token usage would greatly benefit readers.

  • Some missing related work: REFLECT[1] and Causal Agent Replay[2] study attribution through intervention and replay, while ExpeL[3] updates knowledge extracted from trajectory comparisons.

  • There are some important discrepancies between the paper and the released implementation. 1)The implementation does not consistently restart each alternative from the same information state, as described in the paper. Information gathered during earlier attempts can remain in shared memory and influence later alternatives. It also records a changed subgoal as a sibling before executing and verifying it, so the recorded comparison is not necessarily between two executed alternatives starting from the same state. 2) The code counts more cases as redundant loops than the paper’s definition allows. The paper defines a loop as two consecutive execution steps that repeat essentially the same action without obtaining new target information. The implementation also counts candidates that were never executed and actions repeated after intervening steps, without checking whether new target information was obtained.

[1]: Lin, Xiaofeng, et al. "REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces." arXiv preprint arXiv:2606.09071 (2026).

[2]: Shah, Jaineet. "Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures." arXiv preprint arXiv:2606.08275 (2026).

[3]: Zhao, Andrew, et al. "Expel: Llm agents are experiential learners." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.

Questions for authors

See weaknesses.

Reviewer_KsrfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies "epistemic gaps" in LLM-based multi-agent systems, i.e. decisions that were executed correctly but were not supported by the information available when they were made. In a Planner-Executor-Verifier setup, the authors argue that reflecting on the history cannot tell you which level of the decision (subgoal, tool or parameters) was the unsupported one. They propose HIVE, which, when the Verifier flags a failed subgoal, re-executes "sibling" alternatives from the same state in a fixed bottom-up order (parameters first, then tool, then subgoal) and stops at the first success. An LLM then compares the successful and failed siblings and writes what it learns into task-local knowledge for the Planner and the Executor. On six QA and agentic benchmarks HIVE gains around 4.5 points on average over the best of three multi-agent baselines, with shorter trajectories.

Critical strengths
  1. The framing of "which level of the decision lacked information" rather than just "did it fail" is sensible, and the bottom-up escalation is intuitive. The method needs no training and sits on top of a standard Planner-Executor-Verifier loop, so if the gains are real it is easy to adopt.
  2. Fig. 3, which swaps probing for one-shot LLM failure attribution, is a useful controlled comparison and is the strongest evidence in the paper that execution-grounded attribution matters.
Critical weaknesses
  1. HIVE re-executes a failed step up to three more times from the same state, and none of the baselines retries on failure at all (SupervisorAgent for instance is mostly a token-saving method). There is no baseline where the underlying system simply re-executes the failed step at the same iteration cap, or does a Reflexion-style retry from the same state, and no ablation of the probing order or depth. Without these I do not see how the gains can be attributed to the hierarchy or the sibling comparison rather than to extra attempts, and the hierarchy is supposed to be the contribution. The equal-cost claim is an accounting argument with nothing measured, and Clear MAS is missing from the efficiency tables.
  2. Table 7 does not seem possible to produce from the stated protocol. Bamboogle has 375 trajectories across the three runs, so a nonzero loop rate cannot be below about 0.27%, but HIVE is reported at 0.1%, and as far as I can tell several other cells have the same problem. There may be a benign explanation like dropped runs, but the 90% loop-reduction claim rests entirely on this table. Combined with the absence of any variance across three runs on test sets of a few hundred items, I am not confident in the empirical picture.
Questions for authors
  1. What accuracy does Clear MAS get with a naive same-state retry at the same iteration cap, and what does HIVE get with the probing order reversed?
  2. Could you report total LLM calls and tokens per task for every method, including HIVE's reflection and update calls?
  3. What denominators were used for Table 7? Suggestions rather than questions: the paper should be positioned against ExpeL (contrastive reflection over failed and successful trajectories of the same task), tree-search agents such as LATS (branching siblings from a shared state), and DoVer, which is cited but filed under post-hoc diagnosis despite being essentially the Fig. 3 argument applied offline. Verifier reliability should be analyzed, since a false negative writes a wrong "Avoid" rule into the knowledge store. Re-execution from the same state assumes a resettable environment, which should be stated.
Reviewer_PDSWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HIVE, a framework for addressing epistemic gaps in LLM-based multi-agent systems through hierarchical sibling probing and online knowledge updating. When an execution fails, HIVE explores alternative tool parameters, tools, and subgoals bottom-up, compares sibling trajectories to extract decision-specific information, and updates role-specific knowledge. Experiments across six benchmarks and three LLM backbones demonstrate consistent improvements, with HIVE achieving 60.43% average accuracy compared to 55.97% for the strongest baseline, while reducing average trajectory length from 5.83 to 4.45 iterations. Although the results are promising, the experiments do not sufficiently establish whether the improvements arise specifically from identifying and resolving epistemic gaps rather than from structured exploration and repeated repair attempts.

Critical strengths
  • The proposed method addresses inefficient repeated retries through a clear hierarchical intervention strategy that progressively adjusts parameters, tools, and subgoals while preserving useful decisions and enabling early stopping to improve efficiency.

  • The ablation studies demonstrate the complementary benefits of sibling probing and role-specific knowledge updating across all six benchmarks. At the same time, qualitative examples illustrate their effectiveness in query refinement, tool replacement, and subgoal decomposition.

  • The paper provides transparent efficiency accounting by including sibling probes in trajectory length and explicitly accounting for knowledge-update calls in its LLM-call budget analysis.

Critical weaknesses
  • The 4.46-percentage-point improvement is valid when comparing HIVE’s macro average with EPC-AW’s macro average. However, Table 1 defines each dataset’s (\Delta) relative to its best-performing baseline, and the mean of those six displayed improvements is 3.68 percentage points. The average entry and caption currently mix these two comparisons. The abstract should also use “percentage points” rather than “%” for the absolute improvement.

  • The paper should more clearly distinguish HIVE’s novelty from prior interventional debugging methods such as DoVer, particularly regarding hierarchical probing and role-specific knowledge updating. Additionally, the implementation details require clarification, especially how sibling trajectories share the same initial state, handle environmental changes, and incorporate observations from failed executions.

  • The ablation studies do not sufficiently establish the effectiveness of the proposed bottom-up hierarchy or role-specific knowledge representation. Additional comparisons against standard retries, random or reversed intervention orders, and shared-memory updates under matched budgets are needed to isolate their contributions. Furthermore, the LLM-based attribution baseline should be evaluated with comparable repair opportunities and inference resources to ensure a fair comparison.

Questions for authors
  • What evidence distinguishes missing information from other causes of failure? Can the authors provide an annotated sample showing that the inferred information was absent from the original decision context and that the resulting role-specific update is supported? How are success labels made comparable when the subgoal changes?

  • What is the measured inference cost (model calls, input/output tokens, latency, including tool-internal model use and knowledge updating)?

  • What exactly is held fixed across sibling executions? Which observations are available to the alternative generator, how is external state handled, and how is the successful branch incorporated into the continuing task?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.