- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SEER. SEER uses an Information Bottleneck perspective for context management in LLM agents. It does not select evidence by semantic relevance or compression rate. Instead, it selects evidence by how much it reduces the agent's uncertainty about the next environment feedback. It uses marginal entropy reduction per unit token cost as the admission score. The method first formulates context construction as a cost-aware Information Bottleneck objective. It then replaces the latent relevance variable with observable proxy outcome categories. Next, it uses hypothetical actions induced by candidate evidence to predict the proxy outcome distribution. It estimates the action-conditioned entropy from this distribution. This avoids real-environment counterfactual executions for each candidate.
Critical strengths
- The criterion is decision-theoretic. Evidence value is tied to uncertainty reduction over future feedback. This is cleaner than relevance or compression alone.
- It covers four benchmarks. It includes multiple model families and scales. It adds budget-matched comparisons, threshold sweeps, and ablations. Tables 3 and 4 give a credible case that the gains are not just more evidence or more scoring calls.
- The robustness checks are practical. They cover taxonomy granularity, prompt variants, module assignment, and error propagation.
Critical weaknesses
-
ACC on ExCyTIn-Bench is around 25.7. SGC on AppWorld-C is 5.8. Several confidence intervals include zero, including AppWorld-C SGC and SWE-Bench. The paper should separate established gains from directional support more clearly.
-
Table 14 reports roughly 24–28% semantic error for proxy scoring. Table 15 shows non-trivial calibration error. The theory depends on bounded entropy error. But the observed maximum error is around 1.75. It is unclear how tight the threshold rule remains in practice.
-
Inference overhead is not fully resolved. SEER uses many more LLM calls than ACE. On Qwen3-8B it uses 57.4 calls versus 30.0. Wall-clock time is also higher than ACE.
-
Generalization evidence is still narrow. SWE-Bench and WebArena are only tested with Qwen3 backbones. Cross-domain transfer of the evidence taxonomy and the extractor/curator is not systematically studied.
Questions for authors
-
Fixed-budget comparison against ACE and ExpeL. Table 8 shows that SEER uses substantially more LLM calls than ACE (57.4 vs. 30.0 on Qwen3-8B) and higher wall-clock time than ACE. Please add a controlled comparison under a fixed total LLM-call budget and, separately, under a fixed wall-clock budget.
-
Sensitivity of admission to proxy-scoring error. Table 14 reports 24–28% semantic error. Table 15 reports non-trivial calibration error. Admission depends on estimated marginal entropy reduction, not top-1 accuracy. Please add a analyse.
Dear authors,
Thank you for the clarifications. I think the probability handling, threshold, and matched-state interventions address several concerns. Although the discussion deadline is approaching, two points remain important to me.
1. Added value for a strong API agent
In my reading, the practical question in Q4 remains open. I would appreciate any existing comparison of the same strong API agent using standard history handling versus SEER, with matched tools/context and all auxiliary calls included in cost and latency. This requires no RL training of a closed-source model. I understand why stronger models might benefit less, but I am still unsure what added value has been demonstrated for them. If untested, please state that gap and limit the applicability claims accordingly.
2. Revised theoretical scope
I would also appreciate pointers to the revised passages separating the reference-law equivalence from the deployed surrogate, and the local predicted-objective guarantee from task-success guarantees. Please keep online and retrospective entropy distinct. I think consistent revisions here would resolve much of my theoretical concern.