- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces OpenPredictor, a unified harness for general-purpose prediction over heterogeneous answer types (probabilities, numeric arrays, categorical distributions, sets, rankings). Evaluation uses 282 resolved FutureX Past instances across four non-consecutive 2026 weekly bundles and four backbones (DeepSeek V4 Flash, Claude Opus 4.6, Gemini 3.6 Flash, GPT-5.6 Terra), comparing the full workflow against a Direct baseline. However, the comparative experiments are insufficient: the paper does not compare its approach with a sufficiently broad range of agent frameworks and lacks ablation studies.
Critical strengths
The architecture proposed in the paper, particularly its approach of converting a request into a typed task contract, offers a degree of novelty. The validation process is fairly detailed, and the results are reported in a balanced manner. This candor materially enhances the paper’s credibility.
Critical weaknesses
My primary concern with this paper is that the main experiments compare the proposed approach only against direct answers from the LLM, without including ablation studies or other enhancement methods, such as tool use, ReAct, or other capabilities commonly supported by agent frameworks. As a result, the experiments fail to demonstrate the effectiveness of the proposed OpenPredictor, particularly given the complexity of its architecture, which involves three trials, retrieval from six sources, one critic, up to four attempts, gating, and repair mechanisms.
Questions for authors
- Can you provide component ablations and an equal-tool, equal-budget strong baseline?
- Can you evaluate at least one causal or simulation task to demonstrate that OpenPredictor is “general-purpose”?
- How do you bound temporal leakage or model memory given future-dated backbones on 2026 events?
- Where is the code and data?