- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper claims to identify a problem with long-horizon agents - that they commit to a decision before the evidence supports it. They propose a new method called CARE to fix this problem. CARE consists of three main pieces: Consensus Write (handles the initial write), Hypothesis Buffer (holds the unresolved pieces), and Evidence Replay (when new evidence resolves something previously unresolved, this handles it). This is realized through keeping a four dimensional vector with entries for trust, respect, dependency, and conflict, along with a set of pending records. An LLM gives scores for the entries. They threshold to decide if an event is unresolved or not. They introduce a metric, focal displacement, as a way of measuring the changes when the decision is unresolved.
Critical strengths
I would like to preface this by saying that this is not my field of expertise.
(1) The method is new, although similar to others they cite. For example, BeliefMem seems very similar to me, but they don't ever "commit", rather just update probabilities. This method does commit. To me, it does seem better to commit after a certain point, as providing more information even when it's very unlikely seems to be wasting context.
(2) The experiments are evaluated on a large amount of models (in Table 2 they have Claude, DeepSeek, Gemini, GPT, Kimi, and Qwen), indicating that their gain in focal displacement, relative to the DRSM baseline, holds generally.
Critical weaknesses
I would like to preface this by saying that this is not my field of expertise. I have two problems with the main contribution.
(1) The paper says "We identify the failure of committing interpretation-dependent relational effects before the evidence supports them, and introduce focal displacement as a method-neutral measure of state changes at unresolved events". To my knowledge, I do not think identifying that failure is novel. The paper cites BeliefMem, saying that it "preserves alternative conclusions". I agree with this, but the BeliefMem paper in their abstract says that
"By committing to one conclusion and discarding uncertainty, these methods introduce self-reinforcing error: the agent acts on the stored conclusion, never revisits alternatives, and reinforces the conclusion over time. To address this issue, we propose BeliefMem, which shifts the memory paradigm from committing to a single conclusion per observation to retaining multiple candidate conclusions with their probabilities."
It seems that BeliefMem is identifying this failure. I think that their method/solution is different as they never commit, just keep the several options with a probability associated with them, but they do this because they identified the committing before evidence problem.
(2) To the second part of the main contribution, I am not convinced that the metric introduced, focal displacement, is a useful metric. If you're introducing focal displacement as the metric, I think that you need to make a case for it. The two papers cited as related in the introduction use accuracy, F1, BLEU, SR, etc. as metrics they are evaluating against. As other papers in the field use metrics that are scored against a ground truth or downstream outcome, you need to say why your new metric is valid.
To that point, if my understanding is correct, a method that never writes anything would minimize focal displacement.
Questions for authors
(1) Could you clarify if I am misunderstanding something about your main contribution? The authors say that one of their main contributions is identifying "the failure of committing interpretation-dependent relational effects before the evidence supports them". It seems to me that BeliefMem (which is cited in the introduction), and some others, make this identification. Could you explain the difference between what you and they identify?
(2) Could you explain or provide experiment evidence for why focal displacement is a good metric? If my understanding is correct, a method that never writes anything would minimize focal displacement. Without other established metrics accompanying your results, I don't know how to interpret your results. This is my main issue with the paper.
(3) As a smaller point, the paper seems really intent on telling me what experiment was done with the "historical" vs "revised" implementations, first introduced in footnote 1, although this is never defined. Most of the experimental figures and tables in the main paper (all except Figure 5 I believe) use the historical implementation, whatever this means. If this really matters enough to warrant a distinction in the main paper, then shouldn't the experiments be re-run with the revised implementation, or at least have the paper tell us the difference?