- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents Reverie, a multi-agent LLM framework that aims to enable continuous reasoning beyond the usual prompt-response setting. The framework consists of eleven specialized agents that share information through a persistent blackboard. These agents generate thoughts, evaluate ideas, maintain memories and track emotional states. The system also includes an initiative mechanism that allows it to respond without receiving a new user message.
I appreciate the authors releasing their source code for reproducibility. But unfortunately I noticed several inconsistencies between the code and the paper. Some of these directly affect the main claims which makes me less confident in the reported results.
Critical strengths
- I like the motivation behind maintaining a continuous cognitive process across user interactions. Most multi-agent frameworks are organized around individual prompts. So IMO exploring persistent internal deliberation is a good direction to work on. The combination of a shared blackboard, specialized agents and autonomous initiative gives the work a distinct architectural focus.
- The authors evaluate the framework from several perspectives including diversity, coherence, human preference, naturalness and verifiable-task performance. I liked their experiments with different thinker-model sizes. These help provide a broader picture of the frameworks behavior.
Critical weaknesses
-
The paper argues that persistent reasoning across turns is responsible for the improvements. But the BareThinkers ablation removes several components at once (shared workspace, continuous reasoning, feedback between agents etc.). I understand the authors tried to match the number of generated thoughts but this does not necessarily mean the systems receive the same amount of computation or information. So it is difficult to tell which part of the architecture actually drives the improvements. A compute-matched baseline that retains agent interaction but removes persistence would make the central claim more convincing.
-
Out of curiosity, I checked a few source code files of this work. Unfortunately, I found several major inconsistencies between the code and the paper. For example, one of the central claims in main paper and Appendix is that the executive does not receive multi-turn conversation history and that cross-turn information is carried entirely through the persistent cognitive substrate. But in the released
executive.py(lines 120–127),
I can see this line: " conversation = self.blackboard.conversation_history[-3:] "
So the executive appears to receive recent conversation history directly, right?
Similarly, bare_thinkers_batch.py (lines 103–110) also passes conversation history to the executive despite Appendix H describing BareThinkers as having no multi-turn history. So if I am not wrong, this directly contradicts the explanation given in the paper. If the executive can already access previous conversation messages, it becomes difficult to determine how much of the reported 31.4% reduction in cross-turn drift (Section 6.4) actually comes from the cognitive substrate. This is a serious concern because it affects the interpretation of one of the paper's main findings. A comparison where both systems receive exactly the same conversation history would make the claim more convincing(but before that I would expect a clear justification of the issue i mentioned).
- Similarly the paper describes Reverie as a continuously operating(asynchronous multi-agent system with a nominal two-second tick interval). But in
society.py(lines 67–70)
the thinker agents appear to be executed sequentially:
shuffled = thinkers[:]
random.shuffle(shuffled)
for thinker in shuffled:
await thinker.run_tick(self._tick)
If I correctly understand this implementation, each thinker must finish its LLM call before the next thinker can execute. So the actual duration of a tick could be much longer than two seconds depending on the inference time of the underlying model. This could be another issue because several mechanisms in Reverie depend on tick counts (such as attention decay, thought generation and the executive waiting period). If tick duration varies substantially, it is unclear whether the reported timing behavior corresponds to the architecture described in the paper. Could the authors please explain this issue?
-
I dont think generating more diverse thoughts necessarily leads to better responses. Some of these thoughts could be irrelevant or less useful. Although the human preference study provides some supporting evidence, I would have liked to see a more direct analysis of whether higher thought diversity actually correlates with response quality or human preference.
-
The idea of reasoning between user interactions has already been explored in Sleep-time Compute [1]. Although Reverie uses specialized agents and a shared blackboard,, I think the authors should clarify what additional benefits their approach offers over existing background reasoning methods.
Minor:
-
In Figure 1 & 2, the light gray text is difficult to read. Please use a darker color for better visibility.
-
The paper has some inconsistent citation formatting. For instance, in lines 115–123 “Self-Consistency Wang et al., 2023” should be “Self-Consistency (Wang et al., 2023).” The paragraph is also quite dense and difficult to read. I suggest shortening the paragraph and fixing the citation formatting.
[1] Lin et al., "Sleep-time Compute: Beyond Inference Scaling at Test-time," 2025. https://arxiv.org/abs/2504.13171
Questions for authors
- Please refer to Weaknesses where I have discussed my main concerns in detail. I am also happy to be corrected if I have misunderstood any of the points raised in my weaknesses.