acceptodds
Withdrawn from ICLR 2027

OpenPredictor: Harnessing General-Purpose Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

General-purpose prediction must translate open-ended questions into resolvable targets and combine evidence with appropriate estimators across heterogeneous answer types. Language models offer a flexible interface, yet reasoning and tool access alone do not ensure that target semantics are preserved throughout execution. We present OpenPredictor, a unified harness that coordinates this process through explicit task and evidence contracts. Requests become typed targets with declared answer semantics, information cutoffs, and resolution rules. These contracts guide evidence acquisition and restrict estimator selection to compatible methods. The workflow separates model-generated hypotheses from registered numerical operations, retaining units, label identities, and computational dependencies through aggregation and decision construction. Mechanical checks and a model critic guide bounded repair, while linked records make decision derivations inspectable. On FutureX Past, the complete workflow achieves substantial gains over direct answers on several backbones and lower numeric error on jointly answered tasks, although set and ranking accuracy remain challenging. These findings support contract-conditioned execution as a reusable approach to heterogeneous prediction. We will open-source OpenPredictor soon.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Harness, Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZfAvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces OpenPredictor, a unified harness for general-purpose prediction over heterogeneous answer types (probabilities, numeric arrays, categorical distributions, sets, rankings). Evaluation uses 282 resolved FutureX Past instances across four non-consecutive 2026 weekly bundles and four backbones (DeepSeek V4 Flash, Claude Opus 4.6, Gemini 3.6 Flash, GPT-5.6 Terra), comparing the full workflow against a Direct baseline. However, the comparative experiments are insufficient: the paper does not compare its approach with a sufficiently broad range of agent frameworks and lacks ablation studies.

Critical strengths

The architecture proposed in the paper, particularly its approach of converting a request into a typed task contract, offers a degree of novelty. The validation process is fairly detailed, and the results are reported in a balanced manner. This candor materially enhances the paper’s credibility.

Critical weaknesses

My primary concern with this paper is that the main experiments compare the proposed approach only against direct answers from the LLM, without including ablation studies or other enhancement methods, such as tool use, ReAct, or other capabilities commonly supported by agent frameworks. As a result, the experiments fail to demonstrate the effectiveness of the proposed OpenPredictor, particularly given the complexity of its architecture, which involves three trials, retrieval from six sources, one critic, up to four attempts, gating, and repair mechanisms.

Questions for authors
  1. Can you provide component ablations and an equal-tool, equal-budget strong baseline?
  2. Can you evaluate at least one causal or simulation task to demonstrate that OpenPredictor is “general-purpose”?
  3. How do you bound temporal leakage or model memory given future-dated backbones on 2026 events?
  4. Where is the code and data?
Reviewer_3ThbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OpenPredictor, a prediction harness designed for different answer types. The framework defines prediction targets through task contracts, which are used to organize evidence collection and estimator selection. Candidate results generated by the model are processed via typed numerical operations, then checked by Verify, which triggers repair as needed while preserving computational provenance.

On 282 tasks from FutureX Past, the paper compares the full workflow with Direct using four backbone models and supplements the analysis with comparisons against Claude Code and Codex on the corresponding backbones. The full workflow significantly improves L1/L2 scores on three backbones, and all four backbones show a reduction in the median error on numeric tasks answered in both conditions. The content quality of set and ranking predictions remains relatively weak, and Claude’s results are also affected by quota errors.

Critical strengths
  1. This paper proposes a specific workflow design to address the risk of target semantics shifting during prediction. The design in Section 3 links task contracts with evidence requirements and method compatibility, and explicitly handles unit and label information through typed numerical operations. This design clarifies the constraints that must be adhered to at each stage, and computational provenance provides a basis for checking how the results were computed.

  2. The complete workflow achieved significant gains across multiple backbone models. As shown in Table 1, the L1/L2 scores for DeepSeek, Gemini, and GPT improved by 27.17, 22.98, and 12.43 points, respectively; as shown in Figure 5, the median error decreased for each of the four backbones on numeric tasks answered in both conditions, indicating that the improvements were not solely due to changes in the set of answered tasks.

Critical weaknesses
  1. Current experiments do not yet establish how much additional benefit core design elements such as task contracts, method compatibility, and typed numerical operations provide over simpler prediction workflows.

    Section 4.1 primarily compares the single-response "Direct" approach with the full "OpenPredictor" workflow; the two differ substantially in terms of information access and computational effort, and Figure 9 also shows a substantial difference in runtime. Such a comparison cannot distinguish whether the improvement stems from increased resources or from the core design proposed in this paper.

    The general-purpose agent comparison in Section 4.4 also fails to fully address this issue. OpenPredictor’s L1/L2 gain over Claude Code is 2.73 points, with a confidence interval that includes zero, while its non-numeric L3/L4 scores are lower (5.67 vs. 13.58); the comparison with Codex is limited by issues with reference-answer access.

    I would like to see direct comparisons with relevant prediction methods or simplified workflows, using the same backbone with comparable information access and computational budgets. This would help assess the benefits of the specialized prediction harness and the effectiveness of these core design elements.

  2. The current results do not yet provide sufficient support for the framework’s predictive performance across different answer types.

    Of the 153 questions used for the primary L1/L2 metric, 137 were yes/no or multiple-choice questions (Section 4.3, lines 418–420); gains on this metric provide limited evidence of performance on answer types such as sets, entities, and rankings.

    Figure 6 shows that, for the 11 set questions using deterministic scoring, the median score across the four backbones dropped from 65.69 to 39.89; for the 59 ranking questions using judge-based scoring, the corresponding median rose from 2.26 to 7.08 but remained low. This suggests that improvements in output format do not necessarily translate to improvements in the quality of member selection and ranking.

    Furthermore, with only 15 set questions and 11 entity questions in the entire evaluation, it is still difficult to determine whether performance on these question types is stable. I hope the authors will use error examples to explain which aspects have been improved and why existing constraints still fail to resolve certain content errors.

Questions for authors
  1. Section 4.4 and Appendix E.1 mention that some Codex tasks involved reference-answer access. Please explain the basis for this determination and report the number and proportion of affected tasks. The authors state that audit records and the set of excluded tasks are stored in the evaluation artifacts, but these materials are currently not visible in the review interface. Could you provide the relevant records for verification?

  2. The paper implements interfaces for simulation and causal estimation, but the current experiments do not cover these capabilities. Could you provide results on relevant benchmarks or examples of actual runs to demonstrate the functionality of these interfaces?

  3. Section 3.5 mentions that export can retain an earlier format-valid candidate. Please explain the conditions and selection criteria for adopting this strategy, as well as the proportion of tasks in the experiments where this strategy was used. Does the final submission retain the corresponding computational provenance?

  4. If the task contract generated during framing does not accurately reflect the original request, will Verify detect and revise it? Since repair in Section 3.5 primarily returns to Gather or Reason, please explain how such target-definition errors are handled.

  5. For Claude, 95 out of 282 tasks terminated during framing due to quota errors. Appendix B.3 describes call retries and full-episode attempt limits, but does not explain the nature of these errors. What quota limits were reached, and what waiting, retry, or request-control measures were taken? Were these failures related to the framework’s concurrency or call volume? These details would help identify the causes and assess whether request scheduling needs adjustment.

Reviewer_9RpbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces OpenPredictor, a general-purpose prediction agent framework designed to handle heterogeneous forecasting tasks through structured execution contracts. The framework consists of six stages: framing, planning, evidence gathering, reasoning, verification, and publication. Its central design is to represent prediction targets, evidence requirements, estimator compatibility, and answer semantics explicitly, while maintaining computational provenance and supporting verification-driven repair. The authors evaluate OpenPredictor on 282 retrospective FutureX Past tasks across four LLM backbones. Compared with direct LLM predictions, the full system improves deterministic L1/L2 scores on three backbones and substantially reduces median relative error on numerical predictions. Additional analyses examine answer coverage, structured prediction quality, execution failures, and comparisons with general-purpose agent frameworks.

Critical strengths

Using LLM agents for predicting is an interesting topic and the paper provides a coherent framework for organizing heterogeneous prediction workflows.

Critical weaknesses
  1. The main comparison compares with direct LLM. This is very insufficient.
  2. Furthermore, several capabilities emphasized in the framework are not meaningfully evaluated.
  3. The paper also provides little empirical analysis of the specific failure modes these mechanisms are designed to address.
  4. The components of the workflow lack enough motivation. The paper lacks a clearly articulated scientific hypothesis that connects its proposed architecture to the empirical evaluation. The presentation and figures reads like AI-generated.
Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.